Jul 24 13:36:21.833076 ip-10-0-143-193 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 13:36:21.833087 ip-10-0-143-193 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 13:36:21.833094 ip-10-0-143-193 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 13:36:21.833373 ip-10-0-143-193 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 13:36:31.901676 ip-10-0-143-193 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 13:36:31.901694 ip-10-0-143-193 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5ac18ead82c04153b34541190a77a5fc -- Jul 24 13:38:24.657073 ip-10-0-143-193 systemd[1]: Starting Kubernetes Kubelet... Jul 24 13:38:26.691559 ip-10-0-143-193 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:26.691559 ip-10-0-143-193 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 13:38:26.691559 ip-10-0-143-193 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:26.691559 ip-10-0-143-193 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 13:38:26.691559 ip-10-0-143-193 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:26.692623 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.692508 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 13:38:26.697180 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697155 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.697180 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697178 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.697180 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697181 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.697180 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697184 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697187 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697191 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697194 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697196 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697199 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697205 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697209 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697212 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697216 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697218 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697221 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697224 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697227 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697229 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697238 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697241 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697244 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697248 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.697313 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697252 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697255 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697258 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697261 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697264 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697266 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697269 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697272 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697274 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697277 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697280 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697283 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697285 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697288 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697290 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697294 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697297 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697300 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697302 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697305 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.697793 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697308 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697311 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697313 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697316 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697319 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697322 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697325 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697335 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697339 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697341 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697345 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697347 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697350 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697353 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697356 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697358 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697362 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697366 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697368 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697371 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.698283 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697373 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697376 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697379 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697382 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697384 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697387 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697390 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697396 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697398 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697401 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697404 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697407 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697409 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697412 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697415 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697418 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697421 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697423 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697426 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.698831 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697429 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697435 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697438 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697441 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697444 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697447 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697450 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697452 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697455 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697458 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697460 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697463 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697465 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697468 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697470 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697473 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697475 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.697477 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698256 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698264 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.699432 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698267 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698270 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698274 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698277 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698279 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698282 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698285 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698287 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698291 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698294 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698297 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698300 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698302 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698305 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698308 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698311 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698313 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698316 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698318 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.700211 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698321 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698323 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698326 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698328 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698331 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698335 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698339 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698341 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698344 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698347 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698350 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698352 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698355 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698358 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698361 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698364 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698368 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698371 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698375 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.700765 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698378 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698381 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698384 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698387 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698390 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698393 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698395 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698398 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698400 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698403 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698406 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698409 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698411 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698414 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698417 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698419 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698422 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698424 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698427 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698430 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.701235 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698432 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698435 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698438 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698440 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698443 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698446 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698449 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698452 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698455 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698458 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698460 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698463 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698466 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698468 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698471 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698473 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698476 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698481 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698484 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.701733 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698487 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698489 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698491 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698495 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698497 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698500 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698502 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698505 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698508 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698510 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698513 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698516 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698518 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698521 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698523 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698526 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698528 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698531 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698534 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698537 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.702208 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698981 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698987 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698990 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698993 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698995 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.698999 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699001 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699004 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699007 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699009 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699012 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699015 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699017 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699020 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699022 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699080 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699351 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699359 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699364 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699377 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.702712 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699383 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699388 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699393 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699399 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699406 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699411 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699415 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699420 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699424 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699434 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699439 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699447 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699452 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699456 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699460 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699464 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699469 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699474 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699478 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.703202 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699482 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699487 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699496 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699501 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699505 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699510 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699514 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699521 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699527 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699543 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699549 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699556 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699560 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699564 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699574 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699578 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699597 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699602 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699607 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699611 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.703696 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699616 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699620 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699625 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699630 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699636 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699641 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699649 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699653 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699658 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699662 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699666 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699671 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699676 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699680 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699684 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699689 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699695 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699700 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699704 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.704220 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699717 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699721 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699725 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699731 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699767 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699802 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699809 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699814 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699819 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699824 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699829 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699835 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699841 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699846 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699852 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699857 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699867 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699872 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699876 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699880 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.704715 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.699884 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700035 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700047 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700064 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700071 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700077 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700083 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700091 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700098 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700104 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700109 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700119 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700124 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700130 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700135 2581 flags.go:64] FLAG: --cgroup-root="" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700139 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700144 2581 flags.go:64] FLAG: --client-ca-file="" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700152 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700157 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700165 2581 flags.go:64] FLAG: --cluster-domain="" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700175 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700179 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700205 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700232 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 13:38:26.705203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700253 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700258 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700262 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700313 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700440 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700444 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700448 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700452 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700456 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700462 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700465 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700469 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700472 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700475 2581 flags.go:64] FLAG: --enable-server="true" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700478 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700486 2581 flags.go:64] FLAG: --event-burst="100" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700499 2581 flags.go:64] FLAG: --event-qps="50" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700503 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700506 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700509 2581 flags.go:64] FLAG: --eviction-hard="" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700513 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700516 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700519 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700523 2581 flags.go:64] FLAG: --eviction-soft="" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700526 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 13:38:26.705828 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700529 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700532 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700535 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700538 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700541 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700544 2581 flags.go:64] FLAG: --feature-gates="" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700549 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700552 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700556 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700560 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700564 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700568 2581 flags.go:64] FLAG: --help="false" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700571 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-193.ec2.internal" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700575 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700578 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700597 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700603 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700608 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700611 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700614 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700616 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700620 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700623 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700626 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 13:38:26.706454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700629 2581 flags.go:64] FLAG: --kube-reserved="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700632 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700635 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700639 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700641 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700644 2581 flags.go:64] FLAG: --lock-file="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700647 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700650 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700654 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700659 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700662 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700665 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700669 2581 flags.go:64] FLAG: --logging-format="text" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700672 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700675 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700678 2581 flags.go:64] FLAG: --manifest-url="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700682 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700693 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700697 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700702 2581 flags.go:64] FLAG: --max-pods="110" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700705 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700708 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700711 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700714 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700717 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 13:38:26.707070 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700720 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700724 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700732 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700735 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700738 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700741 2581 flags.go:64] FLAG: --pod-cidr="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700744 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700750 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700753 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700756 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700759 2581 flags.go:64] FLAG: --port="10250" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700762 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700765 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01ee71bcb5e4b45f3" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700768 2581 flags.go:64] FLAG: --qos-reserved="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700772 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700775 2581 flags.go:64] FLAG: --register-node="true" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700778 2581 flags.go:64] FLAG: --register-with-taints="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700799 2581 flags.go:64] FLAG: --registry-burst="10" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700804 2581 flags.go:64] FLAG: --registry-qps="5" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700807 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700810 2581 flags.go:64] FLAG: --reserved-memory="" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700814 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700818 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700821 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700824 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 13:38:26.707788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700827 2581 flags.go:64] FLAG: --runonce="false" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700831 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700834 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700837 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700840 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700844 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700847 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700851 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700854 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700857 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700860 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700863 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700866 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700869 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700873 2581 flags.go:64] FLAG: --system-cgroups="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700876 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700882 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700885 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700888 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700896 2581 flags.go:64] FLAG: --tls-min-version="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700899 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700902 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700905 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700909 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700912 2581 flags.go:64] FLAG: --v="2" Jul 24 13:38:26.708505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700917 2581 flags.go:64] FLAG: --version="false" Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700922 2581 flags.go:64] FLAG: --vmodule="" Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700926 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.700930 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701054 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701059 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701062 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701069 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701071 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701075 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701078 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701081 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701083 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701086 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701090 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701094 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701097 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701100 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701102 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701105 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.709149 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701108 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701111 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701114 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701117 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701119 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701122 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701124 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701127 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701130 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701132 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701135 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701137 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701140 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701143 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701145 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701148 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701150 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701153 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701156 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701161 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.709672 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701163 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701168 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701172 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701175 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701178 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701181 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701184 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701187 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701189 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701192 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701203 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701206 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701209 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701211 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701215 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701217 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701220 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701222 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701225 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701228 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.710168 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701230 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701233 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701235 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701238 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701241 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701244 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701246 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701249 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701251 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701254 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701256 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701260 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701263 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701266 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701268 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701271 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701273 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701276 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701278 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701281 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.710682 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701283 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701286 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701288 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701296 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701299 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701302 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701304 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701307 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701310 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701312 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701316 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701319 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701322 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701325 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701328 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701330 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701333 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701335 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701338 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.711173 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701340 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701343 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701346 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.701351 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.702133 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709109 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709128 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709144 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709149 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709212 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709218 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709222 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709226 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709229 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709231 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.711666 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709235 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709238 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709241 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709243 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709246 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709248 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709251 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709254 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709257 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709260 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709262 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709265 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709269 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709273 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709276 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709279 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709282 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709284 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709287 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709290 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.712039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709292 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709295 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709298 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709302 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709305 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709308 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709311 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709314 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709317 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709319 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709322 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709324 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709327 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709330 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709332 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709335 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709337 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709340 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709343 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.712542 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709345 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709348 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709351 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709353 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709356 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709358 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709361 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709363 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709368 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709372 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709375 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709378 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709381 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709385 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709388 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709391 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709395 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709399 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709402 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.713050 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709405 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709408 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709411 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709414 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709416 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709419 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709421 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709424 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709426 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709429 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709431 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709435 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709437 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709440 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709442 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709445 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709448 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709450 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709452 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709455 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.713567 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709457 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709460 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709462 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709465 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709467 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709470 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709472 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709475 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709478 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709481 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709484 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709488 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709490 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709493 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709495 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.714095 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709501 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709638 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709645 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709649 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709651 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709654 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709657 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709660 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709662 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709665 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709668 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709670 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709673 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709675 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709678 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709681 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709684 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709686 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709689 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709692 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:26.714477 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709694 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709696 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709699 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709702 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709705 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709707 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709710 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709713 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709717 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709721 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709724 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709727 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709730 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709733 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709736 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709739 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709741 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709744 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709747 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709751 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:26.714996 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709753 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709756 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709759 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709761 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709764 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709766 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709769 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709771 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709774 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709776 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709779 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709781 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709784 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709786 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709789 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709791 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709794 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709797 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709800 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:26.715504 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709804 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709808 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709811 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709815 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709818 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709821 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709823 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709826 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709828 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709831 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709833 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709836 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709839 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709841 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709843 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709846 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709849 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709851 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709853 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709856 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:26.716049 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709859 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709861 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709864 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709867 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709869 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709872 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709875 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709877 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709879 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709882 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709885 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709888 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709891 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709893 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709896 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709898 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709902 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709904 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709907 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709909 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:26.716520 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:26.709911 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.709918 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.710062 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.712831 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.714069 2581 server.go:1019] "Starting client certificate rotation" Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.714171 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.714239 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 13:38:26.717090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.716968 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 13:38:26.718192 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.718169 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 13:38:26.728633 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.728608 2581 log.go:25] "Validated CRI v1 runtime API" Jul 24 13:38:26.732940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.732920 2581 log.go:25] "Validated CRI v1 image API" Jul 24 13:38:26.733883 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.733862 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 13:38:26.738485 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.738461 2581 fs.go:135] Filesystem UUIDs: map[20a628b6-0c95-4e20-bf27-a74a00549497:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c5a5017c-58d8-4548-bc40-d7fd63cec3a1:/dev/nvme0n1p3] Jul 24 13:38:26.738570 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.738483 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 13:38:26.743651 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.743520 2581 manager.go:216] Machine: {Timestamp:2026-07-24 13:38:26.742283664 +0000 UTC m=+0.338847710 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098403 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2776ac9c57d13b1bfe3137c839e2fa SystemUUID:ec2776ac-9c57-d13b-1bfe-3137c839e2fa BootID:5ac18ead-82c0-4153-b345-41190a77a5fc Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:1f:e8:07:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:1f:e8:07:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:0c:8b:76:df:a8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 13:38:26.743651 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.743642 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 13:38:26.743788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.743735 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 13:38:26.745510 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.745495 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 13:38:26.746315 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.746284 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 13:38:26.746483 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.746320 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 13:38:26.747192 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.747181 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 13:38:26.747224 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.747195 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 13:38:26.747224 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.747208 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 13:38:26.748130 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.748117 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 13:38:26.749315 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.749301 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 13:38:26.751061 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751045 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 24 13:38:26.751212 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751201 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 13:38:26.751241 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751236 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 24 13:38:26.751272 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751250 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 13:38:26.751272 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751266 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 13:38:26.751332 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751277 2581 kubelet.go:398] "Adding apiserver pod source" Jul 24 13:38:26.751332 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751318 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 13:38:26.751740 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751727 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 13:38:26.751804 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.751747 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 13:38:26.754580 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.754562 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 13:38:26.756627 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.756612 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 13:38:26.756700 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.756642 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 13:38:26.758129 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758111 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 13:38:26.758129 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758128 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 13:38:26.758129 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758134 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 13:38:26.758129 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758140 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 13:38:26.758129 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758148 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758154 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758162 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758168 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758174 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758181 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758201 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 13:38:26.758437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.758209 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 13:38:26.759175 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.759164 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 13:38:26.759175 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.759174 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 13:38:26.761392 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.761371 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 13:38:26.761999 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.761977 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 13:38:26.762052 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.761978 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 13:38:26.762536 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.762525 2581 server.go:1268] "Started kubelet" Jul 24 13:38:26.762667 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.762616 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 13:38:26.762747 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.762691 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 13:38:26.762852 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.762640 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 13:38:26.763506 ip-10-0-143-193 systemd[1]: Started Kubernetes Kubelet. Jul 24 13:38:26.764392 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.764340 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 13:38:26.769040 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.769008 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 24 13:38:26.770504 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.770247 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 13:38:26.771549 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.771530 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hzkp" Jul 24 13:38:26.774768 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.774744 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 13:38:26.774854 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.774773 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 13:38:26.775789 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.775665 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 13:38:26.775789 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.775700 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:26.775933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.775791 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 13:38:26.775933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.775806 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 13:38:26.776020 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.775960 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 24 13:38:26.776020 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.775969 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 24 13:38:26.776112 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776101 2581 factory.go:55] Registering systemd factory Jul 24 13:38:26.776159 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776118 2581 factory.go:223] Registration of the systemd container factory successfully Jul 24 13:38:26.776359 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776344 2581 factory.go:153] Registering CRI-O factory Jul 24 13:38:26.776359 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776358 2581 factory.go:223] Registration of the crio container factory successfully Jul 24 13:38:26.776481 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776418 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 13:38:26.776481 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776440 2581 factory.go:103] Registering Raw factory Jul 24 13:38:26.776481 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776452 2581 manager.go:1196] Started watching for new ooms in manager Jul 24 13:38:26.776896 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.776884 2581 manager.go:349] Starting recovery of all containers Jul 24 13:38:26.779022 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.778971 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 13:38:26.779152 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.779131 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 13:38:26.779252 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.779238 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hzkp" Jul 24 13:38:26.780158 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.779047 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-193.ec2.internal.18c53d5f481074b3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-193.ec2.internal,UID:ip-10-0-143-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-193.ec2.internal,},FirstTimestamp:2026-07-24 13:38:26.762495155 +0000 UTC m=+0.359059200,LastTimestamp:2026-07-24 13:38:26.762495155 +0000 UTC m=+0.359059200,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-193.ec2.internal,}" Jul 24 13:38:26.786500 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.786237 2581 manager.go:354] Recovery completed Jul 24 13:38:26.791131 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.791109 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:26.792999 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.792982 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:26.793072 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.793018 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:26.793072 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.793035 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:26.794105 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794085 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 13:38:26.794105 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794098 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:26.794105 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794101 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 13:38:26.794274 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794122 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 24 13:38:26.794842 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794826 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:26.794918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794855 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:26.794918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.794865 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:26.795537 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.795455 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-193.ec2.internal.18c53d5f49e1ed20 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-193.ec2.internal,UID:ip-10-0-143-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-193.ec2.internal,},FirstTimestamp:2026-07-24 13:38:26.793000224 +0000 UTC m=+0.389564282,LastTimestamp:2026-07-24 13:38:26.793000224 +0000 UTC m=+0.389564282,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-193.ec2.internal,}" Jul 24 13:38:26.795878 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.795861 2581 policy_none.go:49] "None policy: Start" Jul 24 13:38:26.795949 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.795882 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 13:38:26.795949 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.795894 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 13:38:26.796995 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.796983 2581 policy_none.go:47] "Start" Jul 24 13:38:26.836308 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836288 2581 manager.go:337] "Starting Device Plugin manager" Jul 24 13:38:26.836436 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.836329 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 13:38:26.836436 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836339 2581 server.go:87] "Starting device plugin registration server" Jul 24 13:38:26.836706 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836692 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 13:38:26.836741 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836709 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 13:38:26.836820 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836798 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 13:38:26.836913 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836896 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 13:38:26.836913 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.836910 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 13:38:26.837526 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.837507 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 13:38:26.837630 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.837564 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:26.908386 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.908339 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 13:38:26.909612 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.909542 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 13:38:26.909686 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.909617 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 13:38:26.909686 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.909654 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 13:38:26.909782 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.909696 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 13:38:26.911790 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.911767 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:26.937524 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.937427 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:26.938725 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.938690 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:26.938725 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.938724 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:26.938898 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.938735 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:26.938898 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.938760 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-193.ec2.internal" Jul 24 13:38:26.947205 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:26.947176 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-193.ec2.internal" Jul 24 13:38:26.947205 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.947211 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-193.ec2.internal\": node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:26.960961 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:26.960928 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.010618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.010563 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal"] Jul 24 13:38:27.010796 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.010685 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:27.012231 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.012215 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:27.012309 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.012249 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:27.012309 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.012266 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:27.013651 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.013634 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:27.013754 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.013738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.013808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.013777 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:27.014443 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014426 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:27.014443 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014436 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:27.014581 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014455 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:27.014581 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014461 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:27.014581 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014468 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:27.014581 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.014471 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:27.015867 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.015853 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.015940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.015880 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:27.016665 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.016649 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:27.016760 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.016679 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:27.016760 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.016689 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:27.041543 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.041515 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-193.ec2.internal\" not found" node="ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.046039 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.046018 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-193.ec2.internal\" not found" node="ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.061685 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.061647 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.078028 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.077995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.078028 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.078027 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.078234 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.078046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9730198ade56c40795706b2e4638cf6a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-193.ec2.internal\" (UID: \"9730198ade56c40795706b2e4638cf6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.162396 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.162357 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.178844 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.178968 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.178968 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9730198ade56c40795706b2e4638cf6a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-193.ec2.internal\" (UID: \"9730198ade56c40795706b2e4638cf6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.178968 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178923 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9730198ade56c40795706b2e4638cf6a-config\") pod \"kube-apiserver-proxy-ip-10-0-143-193.ec2.internal\" (UID: \"9730198ade56c40795706b2e4638cf6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.178968 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178928 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.179111 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.178928 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b05192722bc4feda99ed79b2d21d537-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal\" (UID: \"9b05192722bc4feda99ed79b2d21d537\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.263351 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.263258 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.343871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.343826 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.349483 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.349462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:27.363731 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.363702 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.464288 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.464237 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.564943 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.564852 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.665422 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.665372 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.714010 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.713978 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 13:38:27.714660 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.714176 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 13:38:27.765510 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.765472 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.784661 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.784623 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 13:33:26 +0000 UTC" deadline="2027-12-30 23:04:20.316136921 +0000 UTC" Jul 24 13:38:27.784661 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.784655 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12585h25m52.531485746s" Jul 24 13:38:27.829946 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:27.829904 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9730198ade56c40795706b2e4638cf6a.slice/crio-43ebcedc09841d43bcd887fce77e0127de4800a49cd47d9e1554ffcd3db77dc2 WatchSource:0}: Error finding container 43ebcedc09841d43bcd887fce77e0127de4800a49cd47d9e1554ffcd3db77dc2: Status 404 returned error can't find the container with id 43ebcedc09841d43bcd887fce77e0127de4800a49cd47d9e1554ffcd3db77dc2 Jul 24 13:38:27.830213 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:27.830195 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b05192722bc4feda99ed79b2d21d537.slice/crio-cdec33e1457c1a8c5edca48abcf0e2e6ee8cf67a31a8646a517ed461a2913745 WatchSource:0}: Error finding container cdec33e1457c1a8c5edca48abcf0e2e6ee8cf67a31a8646a517ed461a2913745: Status 404 returned error can't find the container with id cdec33e1457c1a8c5edca48abcf0e2e6ee8cf67a31a8646a517ed461a2913745 Jul 24 13:38:27.834987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.834967 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:38:27.865767 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.865714 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:27.913440 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.913386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" event={"ID":"9730198ade56c40795706b2e4638cf6a","Type":"ContainerStarted","Data":"43ebcedc09841d43bcd887fce77e0127de4800a49cd47d9e1554ffcd3db77dc2"} Jul 24 13:38:27.914364 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:27.914341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" event={"ID":"9b05192722bc4feda99ed79b2d21d537","Type":"ContainerStarted","Data":"cdec33e1457c1a8c5edca48abcf0e2e6ee8cf67a31a8646a517ed461a2913745"} Jul 24 13:38:27.966558 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:27.966513 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-193.ec2.internal\" not found" Jul 24 13:38:28.054567 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.054534 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:28.075839 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.075753 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" Jul 24 13:38:28.087433 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.087402 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 13:38:28.089383 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.089363 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" Jul 24 13:38:28.096156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.096134 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 13:38:28.276833 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.276806 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:28.322867 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.322835 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:28.753376 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.753342 2581 apiserver.go:52] "Watching apiserver" Jul 24 13:38:28.762892 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.762861 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 13:38:28.764664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.764629 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-n7ctg","kube-system/konnectivity-agent-ppq5z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg","openshift-dns/node-resolver-7jz2l","openshift-image-registry/node-ca-zkjz2","openshift-multus/multus-5x7nd","openshift-multus/network-metrics-daemon-tpk5k","kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tkl49","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal","openshift-multus/multus-additional-cni-plugins-qxsvh","openshift-network-diagnostics/network-check-target-mts6c","openshift-network-operator/iptables-alerter-hrhq7","openshift-ovn-kubernetes/ovnkube-node-mbl28"] Jul 24 13:38:28.766409 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.766379 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.766528 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.766465 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:28.767546 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.767528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.768676 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.768660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.769809 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.769792 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 13:38:28.770178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.769908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.770178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.769995 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 13:38:28.770178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.769922 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ngqml\"" Jul 24 13:38:28.770523 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.770507 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2gdlg\"" Jul 24 13:38:28.770641 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.770623 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 13:38:28.770996 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.770866 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.770996 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.770869 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.771393 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.771375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.771847 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.771828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jpd7w\"" Jul 24 13:38:28.771936 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.771827 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.772228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.772208 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.772818 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.772801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:28.772909 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.772864 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:28.773218 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.773202 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.773297 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.773232 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.773347 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.773301 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 13:38:28.773572 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.773556 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9z29k\"" Jul 24 13:38:28.774301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.774279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.775033 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.774989 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 13:38:28.776399 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.776378 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.776399 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.776391 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.776676 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.776656 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-578fx\"" Jul 24 13:38:28.776676 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.776670 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 13:38:28.777374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.777358 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.777457 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.777425 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:28.779000 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.778981 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.780311 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.780294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.781195 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.781064 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.781195 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.781071 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7b7ts\"" Jul 24 13:38:28.781195 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.781092 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.781691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.781673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.782348 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.782331 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.782441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.782427 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2nk7k\"" Jul 24 13:38:28.782658 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.782641 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 13:38:28.782751 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.782641 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 13:38:28.782813 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.782765 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 13:38:28.783234 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.783164 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.783932 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.783445 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.783932 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.783681 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 13:38:28.784127 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.783940 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ppwbc\"" Jul 24 13:38:28.785316 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.785296 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 13:38:28.785420 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.785405 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 13:38:28.785780 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.785763 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 13:38:28.786478 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.786438 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 13:38:28.786478 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.786453 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqtwv\"" Jul 24 13:38:28.786678 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.786621 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 13:38:28.786748 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.786702 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 13:38:28.789073 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.789182 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-sys-fs\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.789182 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789099 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56pw\" (UniqueName: \"kubernetes.io/projected/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kube-api-access-q56pw\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.789182 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-hosts-file\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.789369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:28.789369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbr2w\" (UniqueName: \"kubernetes.io/projected/85012280-6e70-41ee-9d00-58daac481b87-kube-api-access-fbr2w\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.789369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-agent-certs\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.789369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-konnectivity-ca\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.789732 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-socket-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.789821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghj2\" (UniqueName: \"kubernetes.io/projected/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-kube-api-access-xghj2\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.789821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/021b1452-3592-49e5-ac28-7e57e0748712-iptables-alerter-script\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.789918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/021b1452-3592-49e5-ac28-7e57e0748712-host-slash\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.789918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2fr\" (UniqueName: \"kubernetes.io/projected/021b1452-3592-49e5-ac28-7e57e0748712-kube-api-access-lc2fr\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.790013 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-dbus\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.790013 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.789960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-tmp-dir\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.790013 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/821b48f2-7019-4f72-874d-b0435ba169bc-serviceca\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-registration-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-kubelet-config\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790134 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821b48f2-7019-4f72-874d-b0435ba169bc-host\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dmg\" (UniqueName: \"kubernetes.io/projected/821b48f2-7019-4f72-874d-b0435ba169bc-kube-api-access-69dmg\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.790201 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.790200 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-device-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.791082 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.791056 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 13:38:28.836421 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.836398 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v98np" Jul 24 13:38:28.840161 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.840141 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:28.845403 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.845379 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v98np" Jul 24 13:38:28.876768 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.876734 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 13:38:28.891022 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.890985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtd6\" (UniqueName: \"kubernetes.io/projected/4e43cc7e-5db4-4788-ae34-07ba94aad86f-kube-api-access-8wtd6\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.891266 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-system-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891390 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-daemon-config\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891390 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8km\" (UniqueName: \"kubernetes.io/projected/9dc52449-e190-4637-a617-a43cd4feb2be-kube-api-access-nc8km\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.891390 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891334 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-modprobe-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.891390 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-var-lib-kubelet\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.891390 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-os-release\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-cni-binary-copy\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-netns\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-multus\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dc52449-e190-4637-a617-a43cd4feb2be-ovn-node-metrics-cert\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-dbus\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-registration-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-conf-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-systemd-units\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.891664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-device-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56pw\" (UniqueName: \"kubernetes.io/projected/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kube-api-access-q56pw\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-kubernetes\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-dbus\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-lib-modules\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891804 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-registration-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891875 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-device-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-host\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-system-cni-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.891976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-cnibin\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-kubelet\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-kubelet\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.892107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-ovn\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-config\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/021b1452-3592-49e5-ac28-7e57e0748712-iptables-alerter-script\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/021b1452-3592-49e5-ac28-7e57e0748712-host-slash\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2fr\" (UniqueName: \"kubernetes.io/projected/021b1452-3592-49e5-ac28-7e57e0748712-kube-api-access-lc2fr\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-run\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-tmp\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-tmp-dir\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/821b48f2-7019-4f72-874d-b0435ba169bc-serviceca\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-sys-fs\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892457 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-sys\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892495 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-k8s-cni-cncf-io\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.892599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-hostroot\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dmg\" (UniqueName: \"kubernetes.io/projected/821b48f2-7019-4f72-874d-b0435ba169bc-kube-api-access-69dmg\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892619 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-sys-fs\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-netns\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-socket-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-cnibin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-bin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892847 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-tmp-dir\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892856 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-bin\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghj2\" (UniqueName: \"kubernetes.io/projected/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-kube-api-access-xghj2\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892967 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/021b1452-3592-49e5-ac28-7e57e0748712-host-slash\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.892996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-conf\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893010 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-socket-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/821b48f2-7019-4f72-874d-b0435ba169bc-serviceca\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893110 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-slash\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.893400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893113 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/021b1452-3592-49e5-ac28-7e57e0748712-iptables-alerter-script\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-systemd\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-os-release\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfgfv\" (UniqueName: \"kubernetes.io/projected/32852f91-11c0-4e3f-b989-12a10f308796-kube-api-access-nfgfv\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-var-lib-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821b48f2-7019-4f72-874d-b0435ba169bc-host\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.893263 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-log-socket\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-hosts-file\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893343 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/821b48f2-7019-4f72-874d-b0435ba169bc-host\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.893387 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.393365206 +0000 UTC m=+2.989929262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-agent-certs\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-node-log\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-netd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893483 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-hosts-file\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.894179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-env-overrides\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-multus-certs\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893602 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-etc-kubernetes\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzglm\" (UniqueName: \"kubernetes.io/projected/6405943e-85b0-4e60-9bd4-e6f55589d70a-kube-api-access-pzglm\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-etc-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-script-lib\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893714 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-kubelet-config\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893760 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893799 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-kubelet-config\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.893877 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-socket-dir-parent\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-systemd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.894890 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.893993 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25d5e5f6-2725-417c-87d5-dc8234f48fb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.894018 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.394002786 +0000 UTC m=+2.990566838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.894038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbr2w\" (UniqueName: \"kubernetes.io/projected/85012280-6e70-41ee-9d00-58daac481b87-kube-api-access-fbr2w\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.894106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-konnectivity-ca\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.894132 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysconfig\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.894154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-tuned\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.895730 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.894675 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-konnectivity-ca\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.897661 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.897640 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05-agent-certs\") pod \"konnectivity-agent-ppq5z\" (UID: \"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05\") " pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:28.899536 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.899511 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:28.899670 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.899540 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:28.899670 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.899557 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:28.899670 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:28.899643 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.39962539 +0000 UTC m=+2.996189437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:28.900312 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.900293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56pw\" (UniqueName: \"kubernetes.io/projected/25d5e5f6-2725-417c-87d5-dc8234f48fb9-kube-api-access-q56pw\") pod \"aws-ebs-csi-driver-node-82vsg\" (UID: \"25d5e5f6-2725-417c-87d5-dc8234f48fb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:28.900411 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.900298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2fr\" (UniqueName: \"kubernetes.io/projected/021b1452-3592-49e5-ac28-7e57e0748712-kube-api-access-lc2fr\") pod \"iptables-alerter-hrhq7\" (UID: \"021b1452-3592-49e5-ac28-7e57e0748712\") " pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:28.900937 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.900623 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghj2\" (UniqueName: \"kubernetes.io/projected/c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294-kube-api-access-xghj2\") pod \"node-resolver-7jz2l\" (UID: \"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294\") " pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:28.900937 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.900883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dmg\" (UniqueName: \"kubernetes.io/projected/821b48f2-7019-4f72-874d-b0435ba169bc-kube-api-access-69dmg\") pod \"node-ca-zkjz2\" (UID: \"821b48f2-7019-4f72-874d-b0435ba169bc\") " pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:28.902401 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.902379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbr2w\" (UniqueName: \"kubernetes.io/projected/85012280-6e70-41ee-9d00-58daac481b87-kube-api-access-fbr2w\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:28.994826 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-kubelet\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-kubelet\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-ovn\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994923 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-kubelet\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-config\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-kubelet\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.994981 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-ovn\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-run\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-tmp\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-sys\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995267 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-run\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995281 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-sys\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.995432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996766 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-config\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996834 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-binary-copy\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.995327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-k8s-cni-cncf-io\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-k8s-cni-cncf-io\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.996989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-hostroot\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-netns\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997102 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-hostroot\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-cnibin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997165 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-run-netns\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/32852f91-11c0-4e3f-b989-12a10f308796-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997213 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997239 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-bin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-cnibin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997345 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-bin\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997347 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-bin\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.997426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.997398 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-bin\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.999543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.999510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-conf\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.999698 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.999614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-slash\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:28.999698 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.999676 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-systemd\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.999698 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.999693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-conf\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:28.999851 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:28.999702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-os-release\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.000135 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000110 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-slash\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000223 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfgfv\" (UniqueName: \"kubernetes.io/projected/32852f91-11c0-4e3f-b989-12a10f308796-kube-api-access-nfgfv\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.000283 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-var-lib-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000283 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-log-socket\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-node-log\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000329 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-node-log\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-netd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000372 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-systemd\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.000543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000410 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-var-lib-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-env-overrides\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-host-cni-netd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-log-socket\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000523 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-multus-certs\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-etc-kubernetes\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000616 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-multus-certs\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000629 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-etc-kubernetes\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzglm\" (UniqueName: \"kubernetes.io/projected/6405943e-85b0-4e60-9bd4-e6f55589d70a-kube-api-access-pzglm\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-etc-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-script-lib\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-os-release\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.000814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000813 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-env-overrides\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-socket-dir-parent\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-systemd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysconfig\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-socket-dir-parent\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001035 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-run-systemd\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysconfig\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001146 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.000938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-tuned\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001179 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001177 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-etc-openvswitch\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtd6\" (UniqueName: \"kubernetes.io/projected/4e43cc7e-5db4-4788-ae34-07ba94aad86f-kube-api-access-8wtd6\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-system-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001236 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-daemon-config\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8km\" (UniqueName: \"kubernetes.io/projected/9dc52449-e190-4637-a617-a43cd4feb2be-kube-api-access-nc8km\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-modprobe-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001323 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-var-lib-kubelet\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001432 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-modprobe-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-sysctl-d\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-os-release\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-tmp\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-cni-binary-copy\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-netns\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-multus\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dc52449-e190-4637-a617-a43cd4feb2be-ovn-node-metrics-cert\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-conf-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dc52449-e190-4637-a617-a43cd4feb2be-ovnkube-script-lib\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-systemd-units\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.001691 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-kubernetes\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001696 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-run-netns\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-lib-modules\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001738 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-host\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-var-lib-kubelet\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-system-cni-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-cnibin\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-system-cni-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-conf-dir\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001931 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-lib-modules\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001948 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dc52449-e190-4637-a617-a43cd4feb2be-systemd-units\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-multus-daemon-config\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.001979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-host\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-os-release\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-system-cni-dir\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002104 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6405943e-85b0-4e60-9bd4-e6f55589d70a-host-var-lib-cni-multus\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002109 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-kubernetes\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32852f91-11c0-4e3f-b989-12a10f308796-cnibin\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.002574 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.002413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6405943e-85b0-4e60-9bd4-e6f55589d70a-cni-binary-copy\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.004442 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.004393 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dc52449-e190-4637-a617-a43cd4feb2be-ovn-node-metrics-cert\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.007646 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.007622 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e43cc7e-5db4-4788-ae34-07ba94aad86f-etc-tuned\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.008661 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.008552 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfgfv\" (UniqueName: \"kubernetes.io/projected/32852f91-11c0-4e3f-b989-12a10f308796-kube-api-access-nfgfv\") pod \"multus-additional-cni-plugins-qxsvh\" (UID: \"32852f91-11c0-4e3f-b989-12a10f308796\") " pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.008764 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.008729 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8km\" (UniqueName: \"kubernetes.io/projected/9dc52449-e190-4637-a617-a43cd4feb2be-kube-api-access-nc8km\") pod \"ovnkube-node-mbl28\" (UID: \"9dc52449-e190-4637-a617-a43cd4feb2be\") " pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.009211 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.009170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtd6\" (UniqueName: \"kubernetes.io/projected/4e43cc7e-5db4-4788-ae34-07ba94aad86f-kube-api-access-8wtd6\") pod \"tuned-tkl49\" (UID: \"4e43cc7e-5db4-4788-ae34-07ba94aad86f\") " pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.009316 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.009301 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzglm\" (UniqueName: \"kubernetes.io/projected/6405943e-85b0-4e60-9bd4-e6f55589d70a-kube-api-access-pzglm\") pod \"multus-5x7nd\" (UID: \"6405943e-85b0-4e60-9bd4-e6f55589d70a\") " pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.079320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.079288 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:29.087394 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.087364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" Jul 24 13:38:29.096980 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.096952 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7jz2l" Jul 24 13:38:29.101474 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.101455 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkjz2" Jul 24 13:38:29.109564 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.109542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrhq7" Jul 24 13:38:29.117095 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.117077 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkl49" Jul 24 13:38:29.123965 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.123943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" Jul 24 13:38:29.130312 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.130289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5x7nd" Jul 24 13:38:29.136894 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.136874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:29.404674 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.404579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:29.404674 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.404639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.404705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404735 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404747 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404765 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404777 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404803 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404811 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:38:30.404791405 +0000 UTC m=+4.001355460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404830 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:30.404821468 +0000 UTC m=+4.001385508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:29.404921 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:29.404869 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:30.404854371 +0000 UTC m=+4.001418418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:29.706370 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.706332 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821b48f2_7019_4f72_874d_b0435ba169bc.slice/crio-8d6e0dd2498ccc5164c9da8ea328254da8b082d181afd69da780900316ccee0f WatchSource:0}: Error finding container 8d6e0dd2498ccc5164c9da8ea328254da8b082d181afd69da780900316ccee0f: Status 404 returned error can't find the container with id 8d6e0dd2498ccc5164c9da8ea328254da8b082d181afd69da780900316ccee0f Jul 24 13:38:29.707554 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.707523 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc52449_e190_4637_a617_a43cd4feb2be.slice/crio-b106689cded816a72359058e5b99057d2bfafc68c9c067195439993efbb48e9c WatchSource:0}: Error finding container b106689cded816a72359058e5b99057d2bfafc68c9c067195439993efbb48e9c: Status 404 returned error can't find the container with id b106689cded816a72359058e5b99057d2bfafc68c9c067195439993efbb48e9c Jul 24 13:38:29.708319 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.708292 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021b1452_3592_49e5_ac28_7e57e0748712.slice/crio-f4a0cde5426511ca4ec0e2774804b79c9677a8b262ffec2a9af4e9cd5f191a85 WatchSource:0}: Error finding container f4a0cde5426511ca4ec0e2774804b79c9677a8b262ffec2a9af4e9cd5f191a85: Status 404 returned error can't find the container with id f4a0cde5426511ca4ec0e2774804b79c9677a8b262ffec2a9af4e9cd5f191a85 Jul 24 13:38:29.709303 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.709281 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b4c7f5_d393_4fc6_9d7a_ef50e39e5294.slice/crio-6c09747eca7acb82697c4d77ce935a152f193612d20974a9ea0978a792b898a6 WatchSource:0}: Error finding container 6c09747eca7acb82697c4d77ce935a152f193612d20974a9ea0978a792b898a6: Status 404 returned error can't find the container with id 6c09747eca7acb82697c4d77ce935a152f193612d20974a9ea0978a792b898a6 Jul 24 13:38:29.710792 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.710768 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e43cc7e_5db4_4788_ae34_07ba94aad86f.slice/crio-7ded0b8e1c3e35b41b777a836adb1876793c5f8483d8fbcaf36e540f09020507 WatchSource:0}: Error finding container 7ded0b8e1c3e35b41b777a836adb1876793c5f8483d8fbcaf36e540f09020507: Status 404 returned error can't find the container with id 7ded0b8e1c3e35b41b777a836adb1876793c5f8483d8fbcaf36e540f09020507 Jul 24 13:38:29.713943 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.712931 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32852f91_11c0_4e3f_b989_12a10f308796.slice/crio-a08d25a643bc3f744538d5e7c124d5c47b60166291fa2e4f99fd6503eebf9343 WatchSource:0}: Error finding container a08d25a643bc3f744538d5e7c124d5c47b60166291fa2e4f99fd6503eebf9343: Status 404 returned error can't find the container with id a08d25a643bc3f744538d5e7c124d5c47b60166291fa2e4f99fd6503eebf9343 Jul 24 13:38:29.714629 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.714579 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d5e5f6_2725_417c_87d5_dc8234f48fb9.slice/crio-abba347db8b8f70a4766b26bbb94a3ec1d897bebbc8d2fb0da777603970b9583 WatchSource:0}: Error finding container abba347db8b8f70a4766b26bbb94a3ec1d897bebbc8d2fb0da777603970b9583: Status 404 returned error can't find the container with id abba347db8b8f70a4766b26bbb94a3ec1d897bebbc8d2fb0da777603970b9583 Jul 24 13:38:29.715065 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.715043 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode981bc35_bb8b_4ab2_81a6_c0dfb1ff7d05.slice/crio-f19c4d6a0fe9db684f724971ade39b257fcd18569dee1bb66bf74c91ffdda402 WatchSource:0}: Error finding container f19c4d6a0fe9db684f724971ade39b257fcd18569dee1bb66bf74c91ffdda402: Status 404 returned error can't find the container with id f19c4d6a0fe9db684f724971ade39b257fcd18569dee1bb66bf74c91ffdda402 Jul 24 13:38:29.717891 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:38:29.717387 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6405943e_85b0_4e60_9bd4_e6f55589d70a.slice/crio-0138618174ba0d6bb9681455af05cb3b49d39b24174ae702aaea24cacfc0567b WatchSource:0}: Error finding container 0138618174ba0d6bb9681455af05cb3b49d39b24174ae702aaea24cacfc0567b: Status 404 returned error can't find the container with id 0138618174ba0d6bb9681455af05cb3b49d39b24174ae702aaea24cacfc0567b Jul 24 13:38:29.847850 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.847640 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:33:28 +0000 UTC" deadline="2028-03-30 00:10:29.504629253 +0000 UTC" Jul 24 13:38:29.847850 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.847851 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14746h31m59.656785737s" Jul 24 13:38:29.919082 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.919053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkl49" event={"ID":"4e43cc7e-5db4-4788-ae34-07ba94aad86f","Type":"ContainerStarted","Data":"7ded0b8e1c3e35b41b777a836adb1876793c5f8483d8fbcaf36e540f09020507"} Jul 24 13:38:29.920078 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.920058 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jz2l" event={"ID":"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294","Type":"ContainerStarted","Data":"6c09747eca7acb82697c4d77ce935a152f193612d20974a9ea0978a792b898a6"} Jul 24 13:38:29.921068 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.921045 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrhq7" event={"ID":"021b1452-3592-49e5-ac28-7e57e0748712","Type":"ContainerStarted","Data":"f4a0cde5426511ca4ec0e2774804b79c9677a8b262ffec2a9af4e9cd5f191a85"} Jul 24 13:38:29.922766 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.922744 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" event={"ID":"9730198ade56c40795706b2e4638cf6a","Type":"ContainerStarted","Data":"ae1427b4698b78770df30d187c2e84a6e78e9ede3d961666de97956e63c4bc7f"} Jul 24 13:38:29.924084 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.924065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" event={"ID":"9b05192722bc4feda99ed79b2d21d537","Type":"ContainerStarted","Data":"c5dcb3ac221bcde83fd734f5aa3822f127762204e7bb28ac2e43edc1c94ba6ca"} Jul 24 13:38:29.925027 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.925010 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppq5z" event={"ID":"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05","Type":"ContainerStarted","Data":"f19c4d6a0fe9db684f724971ade39b257fcd18569dee1bb66bf74c91ffdda402"} Jul 24 13:38:29.926059 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.926030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"b106689cded816a72359058e5b99057d2bfafc68c9c067195439993efbb48e9c"} Jul 24 13:38:29.927057 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.927039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkjz2" event={"ID":"821b48f2-7019-4f72-874d-b0435ba169bc","Type":"ContainerStarted","Data":"8d6e0dd2498ccc5164c9da8ea328254da8b082d181afd69da780900316ccee0f"} Jul 24 13:38:29.928331 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.928314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" event={"ID":"25d5e5f6-2725-417c-87d5-dc8234f48fb9","Type":"ContainerStarted","Data":"abba347db8b8f70a4766b26bbb94a3ec1d897bebbc8d2fb0da777603970b9583"} Jul 24 13:38:29.929286 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.929269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5x7nd" event={"ID":"6405943e-85b0-4e60-9bd4-e6f55589d70a","Type":"ContainerStarted","Data":"0138618174ba0d6bb9681455af05cb3b49d39b24174ae702aaea24cacfc0567b"} Jul 24 13:38:29.930228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.930210 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerStarted","Data":"a08d25a643bc3f744538d5e7c124d5c47b60166291fa2e4f99fd6503eebf9343"} Jul 24 13:38:29.937019 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:29.936977 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-193.ec2.internal" podStartSLOduration=1.936964949 podStartE2EDuration="1.936964949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:29.936455742 +0000 UTC m=+3.533019798" watchObservedRunningTime="2026-07-24 13:38:29.936964949 +0000 UTC m=+3.533529004" Jul 24 13:38:30.412343 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.412300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:30.412524 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.412356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:30.412524 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.412390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:30.412677 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.412544 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:30.412677 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.412625 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:38:32.412605835 +0000 UTC m=+6.009169881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:30.413049 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413028 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:30.413123 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413091 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:32.413073733 +0000 UTC m=+6.009637770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:30.413187 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413170 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:30.413237 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413189 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:30.413237 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413201 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:30.413237 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.413235 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:32.413223571 +0000 UTC m=+6.009787617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:30.848797 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.848745 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:33:28 +0000 UTC" deadline="2028-03-23 11:32:58.318076048 +0000 UTC" Jul 24 13:38:30.848797 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.848793 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14589h54m27.469288105s" Jul 24 13:38:30.911743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.911711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:30.911931 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.911755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:30.911931 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.911882 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:30.915778 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.915742 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:30.915778 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.911711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:30.915983 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:30.915860 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:30.964892 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.964842 2581 generic.go:356] "Generic (PLEG): container finished" podID="9b05192722bc4feda99ed79b2d21d537" containerID="c5dcb3ac221bcde83fd734f5aa3822f127762204e7bb28ac2e43edc1c94ba6ca" exitCode=0 Jul 24 13:38:30.966088 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:30.966008 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" event={"ID":"9b05192722bc4feda99ed79b2d21d537","Type":"ContainerDied","Data":"c5dcb3ac221bcde83fd734f5aa3822f127762204e7bb28ac2e43edc1c94ba6ca"} Jul 24 13:38:31.985616 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:31.985073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" event={"ID":"9b05192722bc4feda99ed79b2d21d537","Type":"ContainerStarted","Data":"d6f2922ee547dab3ed5234af68a395470936b518a33999ee27de17f034cc86b4"} Jul 24 13:38:32.427376 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.427263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:32.427376 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.427353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:32.427611 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.427392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:32.427611 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.427531 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:32.427611 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.427550 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:32.427611 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.427563 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:32.427808 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.427637 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:36.427617543 +0000 UTC m=+10.024181578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:32.428050 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.428032 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:32.428131 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.428084 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:38:36.428069147 +0000 UTC m=+10.024633184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:32.428190 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.428139 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:32.428190 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.428173 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:36.428161387 +0000 UTC m=+10.024725435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:32.910987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.910907 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:32.911155 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.911043 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:32.911488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.911463 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:32.911611 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.911577 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:32.911667 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:32.911660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:32.911847 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:32.911729 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:34.914510 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:34.914356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:34.914510 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:34.914487 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:34.915054 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:34.914846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:34.915054 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:34.914938 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:34.915054 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:34.914988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:34.915187 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:34.915069 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.463446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.463515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.463550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463604 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463670 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:44.463652051 +0000 UTC m=+18.060216100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463686 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463783 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:38:44.463769965 +0000 UTC m=+18.060333999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463868 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463882 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463892 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:36.463959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.463924 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:44.463914515 +0000 UTC m=+18.060478567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.911035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.911164 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.911551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.911656 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:36.911721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:36.911880 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:36.911793 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:38.910507 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:38.910202 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:38.910507 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:38.910250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:38.910507 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:38.910202 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:38.910507 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:38.910362 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:38.911148 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:38.910510 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:38.911148 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:38.910613 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:40.910486 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:40.910450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:40.910993 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:40.910451 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:40.910993 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:40.910612 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:40.910993 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:40.910450 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:40.910993 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:40.910698 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:40.910993 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:40.910730 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:42.909928 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:42.909888 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:42.910389 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:42.909895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:42.910389 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:42.910037 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:42.910389 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:42.909897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:42.910389 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:42.910095 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:42.910389 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:42.910189 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:44.525016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.524928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:44.525016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.524997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.525030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525065 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525142 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.525125066 +0000 UTC m=+34.121689099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525156 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525197 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525214 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.525197109 +0000 UTC m=+34.121761144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525222 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525236 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:44.525509 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.525292 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.525277441 +0000 UTC m=+34.121841494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:44.910576 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.910489 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:44.910576 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.910490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:44.910809 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.910643 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:44.910809 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:44.910652 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:44.910809 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.910735 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:44.910959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:44.910816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:46.913735 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:46.913704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:46.913735 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:46.913735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:46.914175 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:46.913737 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:46.914175 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:46.913837 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:46.914175 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:46.913914 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:46.914175 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:46.913983 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:48.910122 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:48.909948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:48.910876 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:48.909948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:48.910876 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:48.910209 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:48.910876 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:48.910295 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:48.910876 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:48.910009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:48.910876 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:48.910446 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:49.020966 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.020937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" event={"ID":"25d5e5f6-2725-417c-87d5-dc8234f48fb9","Type":"ContainerStarted","Data":"f7fad13240a2546f0fe9a1149f6ec4a41fc3ccd995df67fdc250b5fdabeea6ba"} Jul 24 13:38:49.022348 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.022324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5x7nd" event={"ID":"6405943e-85b0-4e60-9bd4-e6f55589d70a","Type":"ContainerStarted","Data":"e72f04024238156ce69a48ebefb0213ba0b46bf8506c4c4284944d98aa20ba0e"} Jul 24 13:38:49.023740 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.023698 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="a5fccda8c92c926f552e3e4bf36e5f20bcb280991a142b5f5bb2babc316c0143" exitCode=0 Jul 24 13:38:49.023827 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.023783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"a5fccda8c92c926f552e3e4bf36e5f20bcb280991a142b5f5bb2babc316c0143"} Jul 24 13:38:49.025365 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.025308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkl49" event={"ID":"4e43cc7e-5db4-4788-ae34-07ba94aad86f","Type":"ContainerStarted","Data":"96a3d7ac71884b9944b3a1f845859497a0674dc1761ef9812fd2f016f6611345"} Jul 24 13:38:49.027226 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.027203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7jz2l" event={"ID":"c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294","Type":"ContainerStarted","Data":"b00d735a8e5de2de989a99019354b680cc96af1444f1aac1c7243dccd5ac9a3f"} Jul 24 13:38:49.028847 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.028820 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ppq5z" event={"ID":"e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05","Type":"ContainerStarted","Data":"eea9730989d8f490db2b69108a1fd047389175cc4309fb01a3e18eee4cee6a69"} Jul 24 13:38:49.032224 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.032197 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"9b71a303e67cdf26f324a9b96a1f0c977792a3f2d457014bee2450ea5c6a22ff"} Jul 24 13:38:49.032331 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.032285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"aeb6f96c20d16859507bbe849306f30c21d8be1bf1d0720ed96ddc68690d8f67"} Jul 24 13:38:49.032395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.032340 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"412a556e0241ab98cdbb214342337714f35f1bb35f0612df4c5b116cf9a081c2"} Jul 24 13:38:49.032395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.032389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"61e6a2000102a9b59e3be1f2086fed58fcb45ad768b0418c596b4460c8a07873"} Jul 24 13:38:49.032474 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.032436 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"b9dff88703a953f720d503617aa7d8edfed6a7eee1946c9b25c356afb958082d"} Jul 24 13:38:49.033962 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.033935 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkjz2" event={"ID":"821b48f2-7019-4f72-874d-b0435ba169bc","Type":"ContainerStarted","Data":"1113c0d6c05224169dd170cec889df695a475b1e22ae640696b700c2316a842c"} Jul 24 13:38:49.040712 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.040622 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-193.ec2.internal" podStartSLOduration=21.040603167 podStartE2EDuration="21.040603167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:32.00376757 +0000 UTC m=+5.600331627" watchObservedRunningTime="2026-07-24 13:38:49.040603167 +0000 UTC m=+22.637167215" Jul 24 13:38:49.041301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.041266 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5x7nd" podStartSLOduration=3.3012506999999998 podStartE2EDuration="22.041252581s" totalImagesPullingTime="18.740001881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:49.040428854 +0000 UTC m=+22.636992911" watchObservedRunningTime="2026-07-24 13:38:49.041252581 +0000 UTC m=+22.637816638" Jul 24 13:38:49.057384 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.057336 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zkjz2" podStartSLOduration=4.362472868 podStartE2EDuration="23.057321767s" totalImagesPullingTime="18.694848899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:49.056867034 +0000 UTC m=+22.653431090" watchObservedRunningTime="2026-07-24 13:38:49.057321767 +0000 UTC m=+22.653885821" Jul 24 13:38:49.073177 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.073123 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tkl49" podStartSLOduration=3.380914919 podStartE2EDuration="22.073106155s" totalImagesPullingTime="18.692191236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:49.072998852 +0000 UTC m=+22.669562908" watchObservedRunningTime="2026-07-24 13:38:49.073106155 +0000 UTC m=+22.669670209" Jul 24 13:38:49.110871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.110817 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ppq5z" podStartSLOduration=8.592883839 podStartE2EDuration="23.110799368s" totalImagesPullingTime="14.517915529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:49.110390257 +0000 UTC m=+22.706954563" watchObservedRunningTime="2026-07-24 13:38:49.110799368 +0000 UTC m=+22.707363423" Jul 24 13:38:49.125028 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.124964 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7jz2l" podStartSLOduration=4.43342837 podStartE2EDuration="23.124943275s" totalImagesPullingTime="18.691514905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:49.124184232 +0000 UTC m=+22.720748288" watchObservedRunningTime="2026-07-24 13:38:49.124943275 +0000 UTC m=+22.721507333" Jul 24 13:38:49.875961 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.875916 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 13:38:49.969360 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.969323 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:49.969974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:49.969955 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:50.038808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.038709 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrhq7" event={"ID":"021b1452-3592-49e5-ac28-7e57e0748712","Type":"ContainerStarted","Data":"92957f36163599b7bd1af52e3c4d0fa63ca50036bde9524411b02eedbbfdc830"} Jul 24 13:38:50.042360 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.042326 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"b9005a2328b4caf4b3ee493fe4175fe737a819d7a3e847bd4ae396ec7d80239d"} Jul 24 13:38:50.044522 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.044474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" event={"ID":"25d5e5f6-2725-417c-87d5-dc8234f48fb9","Type":"ContainerStarted","Data":"c519d70592d7993390cabecffdbaa43453c1ca2bbeafcf570811e244a1710550"} Jul 24 13:38:50.045392 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.045371 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:50.045838 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.045819 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ppq5z" Jul 24 13:38:50.057252 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.057197 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hrhq7" podStartSLOduration=4.366839654 podStartE2EDuration="23.057174225s" totalImagesPullingTime="18.690334571s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:50.056600303 +0000 UTC m=+23.653164378" watchObservedRunningTime="2026-07-24 13:38:50.057174225 +0000 UTC m=+23.653738281" Jul 24 13:38:50.849571 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.849490 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T13:38:49.875938567Z","UUID":"1a2f9578-a744-4816-b5be-31ef097d42a4","Handler":null,"Name":"","Endpoint":""} Jul 24 13:38:50.851076 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.851056 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 13:38:50.851184 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.851093 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 13:38:50.910309 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.910281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:50.910473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.910322 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:50.910473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:50.910293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:50.910473 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:50.910418 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:50.910664 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:50.910503 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:50.910664 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:50.910600 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:51.049503 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:51.049467 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" event={"ID":"25d5e5f6-2725-417c-87d5-dc8234f48fb9","Type":"ContainerStarted","Data":"af36f3bd582fbd49dc50dc633ef525c796b9d83337bb14e97e7da06626599deb"} Jul 24 13:38:51.068999 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:51.068947 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82vsg" podStartSLOduration=4.285807586 podStartE2EDuration="25.068929592s" totalImagesPullingTime="20.783122006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:51.068372913 +0000 UTC m=+24.664936968" watchObservedRunningTime="2026-07-24 13:38:51.068929592 +0000 UTC m=+24.665493647" Jul 24 13:38:52.056072 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:52.055838 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"88390953168e1421637069575507a66c0c6eedb753117ec97a97016b5b0f0708"} Jul 24 13:38:52.910842 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:52.910805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:52.910842 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:52.910846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:52.911080 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:52.910805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:52.911080 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:52.910949 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:52.911080 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:52.911025 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:52.911224 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:52.911100 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:54.063124 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.062719 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" event={"ID":"9dc52449-e190-4637-a617-a43cd4feb2be","Type":"ContainerStarted","Data":"5588609ff61c7832f996e1127deca1b627d6e7b8951711983ccc6a44dd079e16"} Jul 24 13:38:54.063124 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.063057 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:54.064667 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.064644 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="a4dde4127ff99e06b6e8dd165388744d2f194b36f19551b2fc0b0bcd46163826" exitCode=0 Jul 24 13:38:54.064728 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.064688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"a4dde4127ff99e06b6e8dd165388744d2f194b36f19551b2fc0b0bcd46163826"} Jul 24 13:38:54.078554 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.078529 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:54.091032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.090983 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" podStartSLOduration=8.338290044 podStartE2EDuration="27.090966134s" totalImagesPullingTime="18.75267609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:54.090925576 +0000 UTC m=+27.687489630" watchObservedRunningTime="2026-07-24 13:38:54.090966134 +0000 UTC m=+27.687530185" Jul 24 13:38:54.910343 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.910304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:54.910604 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.910304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:54.910604 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:54.910444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:54.910604 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:54.910458 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:54.910604 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:54.910535 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:54.910786 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:54.910617 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:55.067291 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.067202 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:55.067291 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.067291 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:55.083944 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.083916 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:38:55.656748 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.656716 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mts6c"] Jul 24 13:38:55.656912 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.656891 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:55.657062 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:55.657006 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:55.657388 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.657369 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7ctg"] Jul 24 13:38:55.657468 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.657460 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:55.657554 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:55.657536 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:55.659204 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.659180 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpk5k"] Jul 24 13:38:55.659321 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:55.659277 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:55.659385 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:55.659365 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:56.070808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:56.070765 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="6f778816ab8929a24626bf79b1d9569b1e41b87bf0a51872a9aeafc4528d4ae9" exitCode=0 Jul 24 13:38:56.071270 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:56.070853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"6f778816ab8929a24626bf79b1d9569b1e41b87bf0a51872a9aeafc4528d4ae9"} Jul 24 13:38:57.910139 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:57.909918 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:57.910547 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:57.909926 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:57.910547 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:57.910181 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:57.910547 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:57.909931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:57.910547 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:57.910249 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:57.910547 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:57.910300 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:38:58.077693 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:58.077657 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="aff35992a190e87160210cd2938b2e7e50bbe2060619fd882fb7404e4eb56926" exitCode=0 Jul 24 13:38:58.077845 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:58.077728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"aff35992a190e87160210cd2938b2e7e50bbe2060619fd882fb7404e4eb56926"} Jul 24 13:38:59.910025 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:59.909987 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:38:59.910025 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:59.910022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:38:59.910829 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:38:59.910026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:38:59.910829 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:59.910122 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mts6c" podUID="b5cf98e2-bf02-4a74-8034-b79578cca326" Jul 24 13:38:59.910829 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:59.910206 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpk5k" podUID="85012280-6e70-41ee-9d00-58daac481b87" Jul 24 13:38:59.910829 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:38:59.910301 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7ctg" podUID="c0b8aee6-bb35-4acf-bcf6-dd596bb799ca" Jul 24 13:39:00.535754 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:00.535715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:39:00.535754 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:00.535764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:00.535788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.535886 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.535901 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.535955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs podName:85012280-6e70-41ee-9d00-58daac481b87 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:32.53594003 +0000 UTC m=+66.132504068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs") pod "network-metrics-daemon-tpk5k" (UID: "85012280-6e70-41ee-9d00-58daac481b87") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.535976 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret podName:c0b8aee6-bb35-4acf-bcf6-dd596bb799ca nodeName:}" failed. No retries permitted until 2026-07-24 13:39:32.535965126 +0000 UTC m=+66.132529159 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret") pod "global-pull-secret-syncer-n7ctg" (UID: "c0b8aee6-bb35-4acf-bcf6-dd596bb799ca") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.535897 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.536002 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:39:00.536024 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.536015 2581 projected.go:196] Error preparing data for projected volume kube-api-access-5k8w7 for pod openshift-network-diagnostics/network-check-target-mts6c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:39:00.536412 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:00.536056 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7 podName:b5cf98e2-bf02-4a74-8034-b79578cca326 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:32.536046496 +0000 UTC m=+66.132610534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5k8w7" (UniqueName: "kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7") pod "network-check-target-mts6c" (UID: "b5cf98e2-bf02-4a74-8034-b79578cca326") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:39:01.260186 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.260155 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-193.ec2.internal" event="NodeReady" Jul 24 13:39:01.260782 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.260297 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 13:39:01.303304 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.303263 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vvskz"] Jul 24 13:39:01.305712 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.305685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.306267 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.306238 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pkzqd"] Jul 24 13:39:01.307779 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.307757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.308030 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.307917 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xl6tf\"" Jul 24 13:39:01.308030 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.307920 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 13:39:01.308030 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.307920 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 13:39:01.310395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.310106 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 13:39:01.310395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.310128 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j9j4b\"" Jul 24 13:39:01.310395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.310188 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 13:39:01.310395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.310188 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 13:39:01.315603 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.315557 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvskz"] Jul 24 13:39:01.317843 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.317785 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkzqd"] Jul 24 13:39:01.441895 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.441862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.441895 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.441897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrr6\" (UniqueName: \"kubernetes.io/projected/fac8536e-3aa4-4112-aeff-5f6b1079ead3-kube-api-access-blrr6\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.442140 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.441928 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda3025-92fd-4593-91b2-9202cf74299d-tmp-dir\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.442140 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.442041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac8536e-3aa4-4112-aeff-5f6b1079ead3-cert\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.442140 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.442082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda3025-92fd-4593-91b2-9202cf74299d-config-volume\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.442140 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.442112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtpx\" (UniqueName: \"kubernetes.io/projected/9eda3025-92fd-4593-91b2-9202cf74299d-kube-api-access-nrtpx\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543243 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda3025-92fd-4593-91b2-9202cf74299d-config-volume\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543243 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtpx\" (UniqueName: \"kubernetes.io/projected/9eda3025-92fd-4593-91b2-9202cf74299d-kube-api-access-nrtpx\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543479 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543479 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrr6\" (UniqueName: \"kubernetes.io/projected/fac8536e-3aa4-4112-aeff-5f6b1079ead3-kube-api-access-blrr6\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.543479 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda3025-92fd-4593-91b2-9202cf74299d-tmp-dir\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543479 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac8536e-3aa4-4112-aeff-5f6b1079ead3-cert\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.543672 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:01.543560 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:01.543672 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:01.543644 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls podName:9eda3025-92fd-4593-91b2-9202cf74299d nodeName:}" failed. No retries permitted until 2026-07-24 13:39:02.043622157 +0000 UTC m=+35.640186205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls") pod "dns-default-vvskz" (UID: "9eda3025-92fd-4593-91b2-9202cf74299d") : secret "dns-default-metrics-tls" not found Jul 24 13:39:01.543758 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eda3025-92fd-4593-91b2-9202cf74299d-tmp-dir\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.543961 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.543944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eda3025-92fd-4593-91b2-9202cf74299d-config-volume\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.548645 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.548617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac8536e-3aa4-4112-aeff-5f6b1079ead3-cert\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.552451 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.552416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtpx\" (UniqueName: \"kubernetes.io/projected/9eda3025-92fd-4593-91b2-9202cf74299d-kube-api-access-nrtpx\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:01.552686 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.552642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrr6\" (UniqueName: \"kubernetes.io/projected/fac8536e-3aa4-4112-aeff-5f6b1079ead3-kube-api-access-blrr6\") pod \"ingress-canary-pkzqd\" (UID: \"fac8536e-3aa4-4112-aeff-5f6b1079ead3\") " pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.625909 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.625867 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkzqd" Jul 24 13:39:01.911075 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.910812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:39:01.911235 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.910830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:01.911276 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.910837 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:39:01.914524 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914506 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 13:39:01.914646 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914530 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 13:39:01.914825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914782 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 13:39:01.914825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914790 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v4t2\"" Jul 24 13:39:01.914825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914778 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnhnl\"" Jul 24 13:39:01.914825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:01.914821 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 13:39:02.018496 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.018467 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkzqd"] Jul 24 13:39:02.022418 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:02.022383 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac8536e_3aa4_4112_aeff_5f6b1079ead3.slice/crio-3567b3b803cb607b3b204fd39b8f0ca5f335070a7b35a6968f01a4a129d68785 WatchSource:0}: Error finding container 3567b3b803cb607b3b204fd39b8f0ca5f335070a7b35a6968f01a4a129d68785: Status 404 returned error can't find the container with id 3567b3b803cb607b3b204fd39b8f0ca5f335070a7b35a6968f01a4a129d68785 Jul 24 13:39:02.047956 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.047923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:02.048099 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:02.048084 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:02.048158 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:02.048152 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls podName:9eda3025-92fd-4593-91b2-9202cf74299d nodeName:}" failed. No retries permitted until 2026-07-24 13:39:03.048133938 +0000 UTC m=+36.644697986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls") pod "dns-default-vvskz" (UID: "9eda3025-92fd-4593-91b2-9202cf74299d") : secret "dns-default-metrics-tls" not found Jul 24 13:39:02.089568 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.089538 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkzqd" event={"ID":"fac8536e-3aa4-4112-aeff-5f6b1079ead3","Type":"ContainerStarted","Data":"3567b3b803cb607b3b204fd39b8f0ca5f335070a7b35a6968f01a4a129d68785"} Jul 24 13:39:02.092714 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.092683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerStarted","Data":"1b007e2ab8684d84e16a0caf499d6ef1df38c1ce5813bca1057a9f1a58142ae3"} Jul 24 13:39:02.276521 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.276480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:02.386703 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.386619 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:02.486962 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.486918 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:02.586525 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.586491 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:02.687426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:02.687392 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:03.056146 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.056057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:03.056304 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:03.056203 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:03.056304 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:03.056270 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls podName:9eda3025-92fd-4593-91b2-9202cf74299d nodeName:}" failed. No retries permitted until 2026-07-24 13:39:05.056250831 +0000 UTC m=+38.652814870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls") pod "dns-default-vvskz" (UID: "9eda3025-92fd-4593-91b2-9202cf74299d") : secret "dns-default-metrics-tls" not found Jul 24 13:39:03.098060 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.098029 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="1b007e2ab8684d84e16a0caf499d6ef1df38c1ce5813bca1057a9f1a58142ae3" exitCode=0 Jul 24 13:39:03.098231 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.098079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"1b007e2ab8684d84e16a0caf499d6ef1df38c1ce5813bca1057a9f1a58142ae3"} Jul 24 13:39:03.454319 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.454284 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:03.517697 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.517659 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:03.735987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.735904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:03.801798 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:03.801763 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:04.104053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:04.103824 2581 generic.go:356] "Generic (PLEG): container finished" podID="32852f91-11c0-4e3f-b989-12a10f308796" containerID="3b40ed9da2e29b64a13318b4af24d9fb8800eb1ca0bf3fa1c966eb9167c5fa4a" exitCode=0 Jul 24 13:39:04.104053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:04.103904 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerDied","Data":"3b40ed9da2e29b64a13318b4af24d9fb8800eb1ca0bf3fa1c966eb9167c5fa4a"} Jul 24 13:39:05.076913 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.076877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:05.077520 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:05.077033 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:05.077520 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:05.077089 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls podName:9eda3025-92fd-4593-91b2-9202cf74299d nodeName:}" failed. No retries permitted until 2026-07-24 13:39:09.077075797 +0000 UTC m=+42.673639831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls") pod "dns-default-vvskz" (UID: "9eda3025-92fd-4593-91b2-9202cf74299d") : secret "dns-default-metrics-tls" not found Jul 24 13:39:05.110409 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.110372 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" event={"ID":"32852f91-11c0-4e3f-b989-12a10f308796","Type":"ContainerStarted","Data":"70b9d7620c50a834016ef22743a33825b77dbc0adcfc782cb65799aa933e5dc0"} Jul 24 13:39:05.111962 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.111936 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkzqd" event={"ID":"fac8536e-3aa4-4112-aeff-5f6b1079ead3","Type":"ContainerStarted","Data":"c37be8c120355efc45fa1d69994121d9da54e21a0b607a1c7ac5865c90b395f1"} Jul 24 13:39:05.134025 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.133971 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qxsvh" podStartSLOduration=17.150869851 podStartE2EDuration="38.133956217s" totalImagesPullingTime="20.983086366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:05.132806946 +0000 UTC m=+38.729371000" watchObservedRunningTime="2026-07-24 13:39:05.133956217 +0000 UTC m=+38.730520296" Jul 24 13:39:05.148924 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.148867 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pkzqd" podStartSLOduration=1.868609977 podStartE2EDuration="4.148847943s" totalImagesPullingTime="2.280237966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:05.147907842 +0000 UTC m=+38.744471896" watchObservedRunningTime="2026-07-24 13:39:05.148847943 +0000 UTC m=+38.745411998" Jul 24 13:39:05.269881 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.269846 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:05.403663 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:05.403564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.374168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.374137 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.474264 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.474197 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.610017 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.609979 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.710137 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.710101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.854525 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.854486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:06.908145 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:06.908080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:07.129003 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.128899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:07.229323 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.229288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:07.609291 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.609258 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl"] Jul 24 13:39:07.643095 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.643059 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl"] Jul 24 13:39:07.643258 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.643177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" Jul 24 13:39:07.645844 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.645822 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 13:39:07.646621 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.646604 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 13:39:07.646753 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.646665 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dcwt4\"" Jul 24 13:39:07.797859 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.797823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5smr\" (UniqueName: \"kubernetes.io/projected/a0b32722-d64a-43ff-b6c7-b19cb05cb829-kube-api-access-j5smr\") pod \"migrator-6865c9967-9xkdl\" (UID: \"a0b32722-d64a-43ff-b6c7-b19cb05cb829\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" Jul 24 13:39:07.898436 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.898363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5smr\" (UniqueName: \"kubernetes.io/projected/a0b32722-d64a-43ff-b6c7-b19cb05cb829-kube-api-access-j5smr\") pod \"migrator-6865c9967-9xkdl\" (UID: \"a0b32722-d64a-43ff-b6c7-b19cb05cb829\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" Jul 24 13:39:07.908229 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.908202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5smr\" (UniqueName: \"kubernetes.io/projected/a0b32722-d64a-43ff-b6c7-b19cb05cb829-kube-api-access-j5smr\") pod \"migrator-6865c9967-9xkdl\" (UID: \"a0b32722-d64a-43ff-b6c7-b19cb05cb829\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" Jul 24 13:39:07.951760 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:07.951727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" Jul 24 13:39:08.096441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.096409 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl"] Jul 24 13:39:08.099383 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:08.099354 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b32722_d64a_43ff_b6c7_b19cb05cb829.slice/crio-9a4a9299305479374bd8d842f2b25e8f7d5f30813beb8bee04fa40200cc2c416 WatchSource:0}: Error finding container 9a4a9299305479374bd8d842f2b25e8f7d5f30813beb8bee04fa40200cc2c416: Status 404 returned error can't find the container with id 9a4a9299305479374bd8d842f2b25e8f7d5f30813beb8bee04fa40200cc2c416 Jul 24 13:39:08.100296 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.100272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:08.120769 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.120735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" event={"ID":"a0b32722-d64a-43ff-b6c7-b19cb05cb829","Type":"ContainerStarted","Data":"9a4a9299305479374bd8d842f2b25e8f7d5f30813beb8bee04fa40200cc2c416"} Jul 24 13:39:08.201432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.201400 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:08.720963 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.720928 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8nsml"] Jul 24 13:39:08.738663 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.738623 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8nsml"] Jul 24 13:39:08.738833 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.738703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:08.741207 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.741169 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 13:39:08.741347 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.741276 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 13:39:08.742143 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.741996 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dsmjs\"" Jul 24 13:39:08.742143 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.742029 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 13:39:08.742143 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.741996 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 13:39:08.757824 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.757770 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:08.824869 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.824837 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:08.905595 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.905548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-cabundle\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:08.905782 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.905608 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-key\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:08.905782 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:08.905636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlw87\" (UniqueName: \"kubernetes.io/projected/a2ed3932-ca32-4e7b-9530-fd45853565cc-kube-api-access-mlw87\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.006374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.006284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-cabundle\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.006374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.006334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-key\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.006374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.006359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlw87\" (UniqueName: \"kubernetes.io/projected/a2ed3932-ca32-4e7b-9530-fd45853565cc-kube-api-access-mlw87\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.007177 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.007148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-cabundle\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.009170 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.009141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a2ed3932-ca32-4e7b-9530-fd45853565cc-signing-key\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.015387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.015363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlw87\" (UniqueName: \"kubernetes.io/projected/a2ed3932-ca32-4e7b-9530-fd45853565cc-kube-api-access-mlw87\") pod \"service-ca-658d7b7c78-8nsml\" (UID: \"a2ed3932-ca32-4e7b-9530-fd45853565cc\") " pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.049223 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.049188 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" Jul 24 13:39:09.107084 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.106980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:09.107243 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:09.107202 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:09.107306 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:09.107269 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls podName:9eda3025-92fd-4593-91b2-9202cf74299d nodeName:}" failed. No retries permitted until 2026-07-24 13:39:17.107250529 +0000 UTC m=+50.703814582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls") pod "dns-default-vvskz" (UID: "9eda3025-92fd-4593-91b2-9202cf74299d") : secret "dns-default-metrics-tls" not found Jul 24 13:39:09.199814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.199784 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-8nsml"] Jul 24 13:39:09.410027 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:09.409948 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ed3932_ca32_4e7b_9530_fd45853565cc.slice/crio-0791d2e49cb16b39088bd3350904dff1cc23ef65a7aad1578a81a70be3a57c09 WatchSource:0}: Error finding container 0791d2e49cb16b39088bd3350904dff1cc23ef65a7aad1578a81a70be3a57c09: Status 404 returned error can't find the container with id 0791d2e49cb16b39088bd3350904dff1cc23ef65a7aad1578a81a70be3a57c09 Jul 24 13:39:09.888956 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:09.888920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:10.128034 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:10.127945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" event={"ID":"a0b32722-d64a-43ff-b6c7-b19cb05cb829","Type":"ContainerStarted","Data":"0159a68b363e9020cfcdce0b3a77fd30a48ed98451633c0220abbaa85c0e9395"} Jul 24 13:39:10.128034 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:10.128007 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" event={"ID":"a0b32722-d64a-43ff-b6c7-b19cb05cb829","Type":"ContainerStarted","Data":"7cb6910cb7cf2c1c84a3d5febc98b733871413c96eab0c7359052d7654a5dd84"} Jul 24 13:39:10.129190 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:10.129165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" event={"ID":"a2ed3932-ca32-4e7b-9530-fd45853565cc","Type":"ContainerStarted","Data":"0791d2e49cb16b39088bd3350904dff1cc23ef65a7aad1578a81a70be3a57c09"} Jul 24 13:39:10.150976 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:10.150923 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9xkdl" podStartSLOduration=1.686023614 podStartE2EDuration="3.150904394s" totalImagesPullingTime="1.46488078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:10.149727589 +0000 UTC m=+43.746291688" watchObservedRunningTime="2026-07-24 13:39:10.150904394 +0000 UTC m=+43.747468426" Jul 24 13:39:12.137174 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:12.137089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" event={"ID":"a2ed3932-ca32-4e7b-9530-fd45853565cc","Type":"ContainerStarted","Data":"44110e1939bfbb6d6130f2070f88bc64b4510981c10376ed8645fcbdf8887842"} Jul 24 13:39:12.157477 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:12.154998 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-8nsml" podStartSLOduration=1.773560389 podStartE2EDuration="4.154967149s" totalImagesPullingTime="2.38140676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:12.153651531 +0000 UTC m=+45.750215586" watchObservedRunningTime="2026-07-24 13:39:12.154967149 +0000 UTC m=+45.751531204" Jul 24 13:39:13.038222 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.038182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.275604 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.275551 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.375995 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.375908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.476252 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.476215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.581887 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.581849 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.676694 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.676660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.778057 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.778015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:13.841516 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:13.841479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:14.557245 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:14.557206 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:14.943355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:14.943317 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:15.343803 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:15.343693 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:15.542293 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:15.542246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:15.939304 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:15.939267 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.337972 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:16.337892 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.742090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:16.742060 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:16.939784 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:16.939715 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:17.143097 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.140580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:17.169927 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.169888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:17.172403 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.172372 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eda3025-92fd-4593-91b2-9202cf74299d-metrics-tls\") pod \"dns-default-vvskz\" (UID: \"9eda3025-92fd-4593-91b2-9202cf74299d\") " pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:17.220342 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.220302 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:17.371974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.371928 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:17.378314 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.378279 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvskz"] Jul 24 13:39:17.381871 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:17.381829 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eda3025_92fd_4593_91b2_9202cf74299d.slice/crio-6dd2055e5d3f89bca8c03df39ee6d51ff4bc469260b131dfac3daf1763370978 WatchSource:0}: Error finding container 6dd2055e5d3f89bca8c03df39ee6d51ff4bc469260b131dfac3daf1763370978: Status 404 returned error can't find the container with id 6dd2055e5d3f89bca8c03df39ee6d51ff4bc469260b131dfac3daf1763370978 Jul 24 13:39:17.543505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.543464 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:17.607866 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.607834 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:17.956426 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:17.956389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:18.153454 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:18.153422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvskz" event={"ID":"9eda3025-92fd-4593-91b2-9202cf74299d","Type":"ContainerStarted","Data":"6dd2055e5d3f89bca8c03df39ee6d51ff4bc469260b131dfac3daf1763370978"} Jul 24 13:39:18.544151 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:18.544119 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.156411 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:19.156370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.358988 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:19.358957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.450177 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:19.450139 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.756409 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:19.756376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:19.967969 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:19.967817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:20.161301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.161268 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvskz" event={"ID":"9eda3025-92fd-4593-91b2-9202cf74299d","Type":"ContainerStarted","Data":"9f793ac968591c3c9ec738d0dd69ac79226cb158d6513a7d25ff257a6684dc5c"} Jul 24 13:39:20.161744 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.161322 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvskz" event={"ID":"9eda3025-92fd-4593-91b2-9202cf74299d","Type":"ContainerStarted","Data":"9b0513b0809f9220f59adf2bdbbdc0b98f14479163927057f4d01cf776b12b8b"} Jul 24 13:39:20.161744 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.161435 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:20.182163 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.182107 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vvskz" podStartSLOduration=16.962141956 podStartE2EDuration="19.182091243s" totalImagesPullingTime="2.219949287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:20.181447451 +0000 UTC m=+53.778011506" watchObservedRunningTime="2026-07-24 13:39:20.182091243 +0000 UTC m=+53.778655295" Jul 24 13:39:20.350524 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.350444 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:20.962485 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:20.962449 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.355387 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:21.355240 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.443642 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:21.443579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:21.758371 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:21.758321 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:22.174776 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:22.174745 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:22.612290 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:22.612198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:22.949725 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:22.949691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:23.163484 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:23.163443 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:23.388953 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:23.388872 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:27.085824 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:27.085795 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mbl28" Jul 24 13:39:30.166734 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:30.166702 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vvskz" Jul 24 13:39:31.676670 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.676631 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n"] Jul 24 13:39:31.679509 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.679479 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns"] Jul 24 13:39:31.679718 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.679648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.681792 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.681770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.685301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.685278 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 13:39:31.685533 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.685517 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 13:39:31.686298 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686276 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 13:39:31.686433 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686348 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 13:39:31.686504 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686437 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 13:39:31.686504 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686366 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-87f4q\"" Jul 24 13:39:31.686627 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686390 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 13:39:31.686627 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686416 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 13:39:31.686627 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.686375 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 13:39:31.694673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.694645 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n"] Jul 24 13:39:31.695323 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.695302 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns"] Jul 24 13:39:31.779773 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.779729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk2d\" (UniqueName: \"kubernetes.io/projected/720bcc34-6e21-4e62-b91d-dc41369b0ac3-kube-api-access-jmk2d\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.779989 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.779839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstm7\" (UniqueName: \"kubernetes.io/projected/c98513f0-3d2b-4a51-b794-4ceefef61230-kube-api-access-rstm7\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.779989 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.779915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720bcc34-6e21-4e62-b91d-dc41369b0ac3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.779989 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.779948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.779989 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.779985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.780168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.780012 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.780168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.780047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98513f0-3d2b-4a51-b794-4ceefef61230-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.780168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.780120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.788294 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.788256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:31.867182 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.867148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:31.880450 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.880450 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.880688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880476 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.880688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98513f0-3d2b-4a51-b794-4ceefef61230-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.880688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.880856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880827 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk2d\" (UniqueName: \"kubernetes.io/projected/720bcc34-6e21-4e62-b91d-dc41369b0ac3-kube-api-access-jmk2d\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.880916 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880873 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstm7\" (UniqueName: \"kubernetes.io/projected/c98513f0-3d2b-4a51-b794-4ceefef61230-kube-api-access-rstm7\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.880969 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.880954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720bcc34-6e21-4e62-b91d-dc41369b0ac3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.881607 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.881560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/720bcc34-6e21-4e62-b91d-dc41369b0ac3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.883258 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.883233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.883611 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.883574 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-ca\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.883685 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.883663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-hub\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.883735 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.883715 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/720bcc34-6e21-4e62-b91d-dc41369b0ac3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.883847 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.883833 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c98513f0-3d2b-4a51-b794-4ceefef61230-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.899299 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.899273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk2d\" (UniqueName: \"kubernetes.io/projected/720bcc34-6e21-4e62-b91d-dc41369b0ac3-kube-api-access-jmk2d\") pod \"cluster-proxy-proxy-agent-6bbf76d885-28rns\" (UID: \"720bcc34-6e21-4e62-b91d-dc41369b0ac3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:31.899456 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.899388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstm7\" (UniqueName: \"kubernetes.io/projected/c98513f0-3d2b-4a51-b794-4ceefef61230-kube-api-access-rstm7\") pod \"managed-serviceaccount-addon-agent-cdc7594b4-7289n\" (UID: \"c98513f0-3d2b-4a51-b794-4ceefef61230\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:31.943032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:31.942941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.006370 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.006327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" Jul 24 13:39:32.010933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.010905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" Jul 24 13:39:32.047508 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.047393 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.143305 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.143271 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n"] Jul 24 13:39:32.145758 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.145731 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.147353 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:32.147326 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc98513f0_3d2b_4a51_b794_4ceefef61230.slice/crio-9c919b1ee3b41ceb2f4d539bb372e0aee057540d531bb4d1cc45066969f8b206 WatchSource:0}: Error finding container 9c919b1ee3b41ceb2f4d539bb372e0aee057540d531bb4d1cc45066969f8b206: Status 404 returned error can't find the container with id 9c919b1ee3b41ceb2f4d539bb372e0aee057540d531bb4d1cc45066969f8b206 Jul 24 13:39:32.162069 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.162037 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns"] Jul 24 13:39:32.165459 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:32.165433 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod720bcc34_6e21_4e62_b91d_dc41369b0ac3.slice/crio-3624f026ef9c0bdae09ebff6c088409ca9a734f74433c859ecce0ea6f9e80411 WatchSource:0}: Error finding container 3624f026ef9c0bdae09ebff6c088409ca9a734f74433c859ecce0ea6f9e80411: Status 404 returned error can't find the container with id 3624f026ef9c0bdae09ebff6c088409ca9a734f74433c859ecce0ea6f9e80411 Jul 24 13:39:32.199469 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.199355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" event={"ID":"720bcc34-6e21-4e62-b91d-dc41369b0ac3","Type":"ContainerStarted","Data":"3624f026ef9c0bdae09ebff6c088409ca9a734f74433c859ecce0ea6f9e80411"} Jul 24 13:39:32.200421 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.200398 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" event={"ID":"c98513f0-3d2b-4a51-b794-4ceefef61230","Type":"ContainerStarted","Data":"9c919b1ee3b41ceb2f4d539bb372e0aee057540d531bb4d1cc45066969f8b206"} Jul 24 13:39:32.225960 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.225929 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.313259 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.313224 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.409844 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.409812 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.505764 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.505673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:32.587439 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.587391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:39:32.587439 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.587441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:32.587687 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.587575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:39:32.590123 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.590098 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 13:39:32.590789 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.590770 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 13:39:32.590842 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.590776 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 13:39:32.600815 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.600788 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 13:39:32.600933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.600854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0b8aee6-bb35-4acf-bcf6-dd596bb799ca-original-pull-secret\") pod \"global-pull-secret-syncer-n7ctg\" (UID: \"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca\") " pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:39:32.600933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.600867 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85012280-6e70-41ee-9d00-58daac481b87-metrics-certs\") pod \"network-metrics-daemon-tpk5k\" (UID: \"85012280-6e70-41ee-9d00-58daac481b87\") " pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:39:32.611171 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.611142 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8w7\" (UniqueName: \"kubernetes.io/projected/b5cf98e2-bf02-4a74-8034-b79578cca326-kube-api-access-5k8w7\") pod \"network-check-target-mts6c\" (UID: \"b5cf98e2-bf02-4a74-8034-b79578cca326\") " pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:32.824887 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.824800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7ctg" Jul 24 13:39:32.844603 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.844562 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnhnl\"" Jul 24 13:39:32.852728 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.852452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:32.853616 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.853559 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v4t2\"" Jul 24 13:39:32.862449 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:32.862418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpk5k" Jul 24 13:39:33.038110 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.037990 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7ctg"] Jul 24 13:39:33.051649 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:33.049507 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b8aee6_bb35_4acf_bcf6_dd596bb799ca.slice/crio-301e3a179521f475d7007cdf8b9618a551ac4f89e380f879560fb9567ca4ef26 WatchSource:0}: Error finding container 301e3a179521f475d7007cdf8b9618a551ac4f89e380f879560fb9567ca4ef26: Status 404 returned error can't find the container with id 301e3a179521f475d7007cdf8b9618a551ac4f89e380f879560fb9567ca4ef26 Jul 24 13:39:33.073285 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.073262 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mts6c"] Jul 24 13:39:33.077495 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:33.077415 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cf98e2_bf02_4a74_8034_b79578cca326.slice/crio-de59d88579791c9904d2163359f9a7d219ea7943966d9a6799d48c9d158cd915 WatchSource:0}: Error finding container de59d88579791c9904d2163359f9a7d219ea7943966d9a6799d48c9d158cd915: Status 404 returned error can't find the container with id de59d88579791c9904d2163359f9a7d219ea7943966d9a6799d48c9d158cd915 Jul 24 13:39:33.106123 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.106089 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpk5k"] Jul 24 13:39:33.113681 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:33.113636 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85012280_6e70_41ee_9d00_58daac481b87.slice/crio-75103b82373d45902667b665359bd0cec9fe73cec29cd0a350cf2d5545ffa303 WatchSource:0}: Error finding container 75103b82373d45902667b665359bd0cec9fe73cec29cd0a350cf2d5545ffa303: Status 404 returned error can't find the container with id 75103b82373d45902667b665359bd0cec9fe73cec29cd0a350cf2d5545ffa303 Jul 24 13:39:33.209780 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.209741 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpk5k" event={"ID":"85012280-6e70-41ee-9d00-58daac481b87","Type":"ContainerStarted","Data":"75103b82373d45902667b665359bd0cec9fe73cec29cd0a350cf2d5545ffa303"} Jul 24 13:39:33.211504 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.211472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n7ctg" event={"ID":"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca","Type":"ContainerStarted","Data":"301e3a179521f475d7007cdf8b9618a551ac4f89e380f879560fb9567ca4ef26"} Jul 24 13:39:33.213441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.213383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mts6c" event={"ID":"b5cf98e2-bf02-4a74-8034-b79578cca326","Type":"ContainerStarted","Data":"de59d88579791c9904d2163359f9a7d219ea7943966d9a6799d48c9d158cd915"} Jul 24 13:39:33.631663 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.631458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:33.719624 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:33.719508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:36.416243 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:36.416201 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:36.513545 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:36.513507 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:38.413716 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:38.413626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:38.513724 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:38.513671 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:38.614612 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:38.614564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:38.737303 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:38.737257 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.137122 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.137043 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cqkcf"] Jul 24 13:39:39.139405 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.139380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.142296 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.142260 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 13:39:39.142595 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.142543 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 13:39:39.142725 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.142639 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 13:39:39.142905 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.142883 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 13:39:39.143518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.143501 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbx6j\"" Jul 24 13:39:39.153207 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.153180 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cqkcf"] Jul 24 13:39:39.166271 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.166239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.237695 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.237658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.239673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.239647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.239814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.239710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.239814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.239750 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29bf\" (UniqueName: \"kubernetes.io/projected/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-api-access-m29bf\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.239814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.239776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-crio-socket\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.239814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.239793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-data-volume\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.304760 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.304719 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.340348 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29bf\" (UniqueName: \"kubernetes.io/projected/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-api-access-m29bf\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-crio-socket\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-data-volume\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.340453 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-crio-socket\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.340552 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.340537 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:39.840512815 +0000 UTC m=+73.437076871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:39.340902 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.340626 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.340902 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.340671 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:39.84065783 +0000 UTC m=+73.437221878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.340902 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.340763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-data-volume\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.351150 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.351116 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29bf\" (UniqueName: \"kubernetes.io/projected/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-api-access-m29bf\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.711512 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.711473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.843054 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.843013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.843311 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.843095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:39.843311 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.843191 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:39.843311 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.843216 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.843311 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.843279 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:40.843235348 +0000 UTC m=+74.439799398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:39.843311 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:39.843299 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:40.843289862 +0000 UTC m=+74.439853898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.843661 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.843366 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:39.943818 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:39.943783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.237662 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.237629 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.240414 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.240389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mts6c" event={"ID":"b5cf98e2-bf02-4a74-8034-b79578cca326","Type":"ContainerStarted","Data":"51d72486887a850743964db88dad6ba0f3ab5c7ff836b9ea866e2cd1156901b7"} Jul 24 13:39:40.240575 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.240467 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:39:40.242360 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.242338 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" event={"ID":"720bcc34-6e21-4e62-b91d-dc41369b0ac3","Type":"ContainerStarted","Data":"24b00b74232168767cd15409c1f5e1b05f8c4e46a624c866cd3d653988503a88"} Jul 24 13:39:40.242485 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.242375 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" event={"ID":"720bcc34-6e21-4e62-b91d-dc41369b0ac3","Type":"ContainerStarted","Data":"d3bab57998542c63828219c603a3fdb8f779b50936d42a28452f807463be5a88"} Jul 24 13:39:40.242485 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.242391 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" event={"ID":"720bcc34-6e21-4e62-b91d-dc41369b0ac3","Type":"ContainerStarted","Data":"3521a23c59d040a49ba5249abdda5e22cca775b458182214dafbf7278770ce38"} Jul 24 13:39:40.244087 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.244061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpk5k" event={"ID":"85012280-6e70-41ee-9d00-58daac481b87","Type":"ContainerStarted","Data":"3fee1628c49f87de0c3a89f5725994cb92cf7c16736c04b7e65e91f11faab8b0"} Jul 24 13:39:40.244197 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.244119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpk5k" event={"ID":"85012280-6e70-41ee-9d00-58daac481b87","Type":"ContainerStarted","Data":"850cc6f49e49cd494cdafb024107ced63b8a130b3fb0f891db8717a8696aa6b4"} Jul 24 13:39:40.245436 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.245412 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n7ctg" event={"ID":"c0b8aee6-bb35-4acf-bcf6-dd596bb799ca","Type":"ContainerStarted","Data":"0c707b4ddb88be989b41a2d4441afa1a26b296f0bca24df7d1223824b6266444"} Jul 24 13:39:40.246748 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.246728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" event={"ID":"c98513f0-3d2b-4a51-b794-4ceefef61230","Type":"ContainerStarted","Data":"cca0359efba884316f82b57c66937a7d739eb82e434b737e71bcc20da70ce6a2"} Jul 24 13:39:40.257349 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.257290 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mts6c" podStartSLOduration=67.761926811 podStartE2EDuration="1m14.257271472s" totalImagesPullingTime="6.495344661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:40.25715441 +0000 UTC m=+73.853718465" watchObservedRunningTime="2026-07-24 13:39:40.257271472 +0000 UTC m=+73.853835529" Jul 24 13:39:40.274713 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.274640 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n7ctg" podStartSLOduration=66.72800892 podStartE2EDuration="1m13.274618331s" totalImagesPullingTime="6.546609411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:40.274224843 +0000 UTC m=+73.870788899" watchObservedRunningTime="2026-07-24 13:39:40.274618331 +0000 UTC m=+73.871182383" Jul 24 13:39:40.313219 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.313166 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cdc7594b4-7289n" podStartSLOduration=1.903922354 podStartE2EDuration="9.313150625s" totalImagesPullingTime="7.409228271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:40.312405545 +0000 UTC m=+73.908969593" watchObservedRunningTime="2026-07-24 13:39:40.313150625 +0000 UTC m=+73.909714680" Jul 24 13:39:40.313571 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.313551 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bbf76d885-28rns" podStartSLOduration=1.9030839510000002 podStartE2EDuration="9.313543995s" totalImagesPullingTime="7.410460044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:40.293989867 +0000 UTC m=+73.890553923" watchObservedRunningTime="2026-07-24 13:39:40.313543995 +0000 UTC m=+73.910108049" Jul 24 13:39:40.338382 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.338322 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tpk5k" podStartSLOduration=67.897814783 podStartE2EDuration="1m14.338302562s" totalImagesPullingTime="6.440487779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:40.337412078 +0000 UTC m=+73.933976133" watchObservedRunningTime="2026-07-24 13:39:40.338302562 +0000 UTC m=+73.934866617" Jul 24 13:39:40.351102 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.351068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.443377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.443336 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.507940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.507851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.514958 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.514933 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvskz_9eda3025-92fd-4593-91b2-9202cf74299d/dns/0.log" Jul 24 13:39:40.653636 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.653600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.717864 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.717835 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvskz_9eda3025-92fd-4593-91b2-9202cf74299d/kube-rbac-proxy/0.log" Jul 24 13:39:40.754013 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.753979 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.848912 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.848812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:40.848912 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.848884 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:40.849139 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:40.848954 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:40.849139 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:40.848969 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:40.849139 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:40.849025 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:42.849010078 +0000 UTC m=+76.445574111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:40.849139 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:40.849041 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:42.849034783 +0000 UTC m=+76.445598815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:40.854251 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.854223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:40.914185 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.914154 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7jz2l_c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294/dns-node-resolver/0.log" Jul 24 13:39:40.956115 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:40.956088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.057408 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.057368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.274146 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.274107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.371007 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.370975 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.698950 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.698918 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.780837 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.780802 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:41.883694 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:41.883665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:42.514940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:42.514913 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkjz2_821b48f2-7019-4f72-874d-b0435ba169bc/node-ca/0.log" Jul 24 13:39:42.826530 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:42.826413 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:42.863157 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:42.863122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:42.863261 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:42.863179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:42.863261 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:42.863251 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:42.863340 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:42.863282 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:42.863340 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:42.863312 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:46.863297447 +0000 UTC m=+80.459861480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:42.863340 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:42.863326 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls podName:3834f8c9-b9c9-4128-bd5e-cfd2a694e56e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:46.863319909 +0000 UTC m=+80.459883943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cqkcf" (UID: "3834f8c9-b9c9-4128-bd5e-cfd2a694e56e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:42.926837 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:42.926800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:43.117425 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:43.117347 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkzqd_fac8536e-3aa4-4112-aeff-5f6b1079ead3/serve-healthcheck-canary/0.log" Jul 24 13:39:43.574711 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:43.574672 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:44.826920 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:44.826883 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.049984 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.049950 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.107001 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.106909 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vrwt7"] Jul 24 13:39:45.110665 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.110638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.112953 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.112930 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 13:39:45.113084 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.112986 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbctl\"" Jul 24 13:39:45.113322 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.113299 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 13:39:45.114125 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.114097 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 13:39:45.114228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.114127 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 13:39:45.114370 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.114346 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 13:39:45.114475 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.114450 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 13:39:45.133266 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.133231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.179979 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.179941 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-textfile\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.179979 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.179982 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-wtmp\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-metrics-client-ca\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180078 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-kube-api-access-t5hgs\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-sys\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.180246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.180220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-root\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.237766 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.237735 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.280704 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-wtmp\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.280704 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-metrics-client-ca\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-kube-api-access-t5hgs\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-sys\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.280934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-sys\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-root\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281373 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:45.281334 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 13:39:45.281373 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281363 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-textfile\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281396 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-metrics-client-ca\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281473 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:39:45.281413 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls podName:0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:45.781394602 +0000 UTC m=+79.377958640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls") pod "node-exporter-vrwt7" (UID: "0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51") : secret "node-exporter-tls" not found Jul 24 13:39:45.281473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281430 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-accelerators-collector-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281463 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-root\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281709 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-wtmp\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.281709 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.281611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-textfile\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.283913 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.283889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.292386 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.292353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hgs\" (UniqueName: \"kubernetes.io/projected/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-kube-api-access-t5hgs\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.343249 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.343209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.443871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.443835 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.540274 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.540237 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.639802 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.639771 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:45.786504 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.786422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:45.788705 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:45.788683 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51-node-exporter-tls\") pod \"node-exporter-vrwt7\" (UID: \"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51\") " pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:46.021548 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.021511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vrwt7" Jul 24 13:39:46.033157 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:46.033124 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dea3f02_6d7c_4d4d_9c8b_9cc9c6282f51.slice/crio-fb0dbfa138b58028f9e9a13df374528a0c63594311177f44b7978363649f81c8 WatchSource:0}: Error finding container fb0dbfa138b58028f9e9a13df374528a0c63594311177f44b7978363649f81c8: Status 404 returned error can't find the container with id fb0dbfa138b58028f9e9a13df374528a0c63594311177f44b7978363649f81c8 Jul 24 13:39:46.073241 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.073196 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:46.154614 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.154565 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:46.273134 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.273095 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vrwt7" event={"ID":"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51","Type":"ContainerStarted","Data":"fb0dbfa138b58028f9e9a13df374528a0c63594311177f44b7978363649f81c8"} Jul 24 13:39:46.374242 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.374152 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:46.895411 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.895366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:46.895647 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.895430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:46.896093 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.896063 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:46.898417 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.898356 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3834f8c9-b9c9-4128-bd5e-cfd2a694e56e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cqkcf\" (UID: \"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e\") " pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:46.950712 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:46.950670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cqkcf" Jul 24 13:39:47.105442 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.105304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cqkcf"] Jul 24 13:39:47.108452 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:39:47.108425 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3834f8c9_b9c9_4128_bd5e_cfd2a694e56e.slice/crio-adcfa710c231805411977aaeba54bf1f2078cf927711f2921132b77dd7b0b5b6 WatchSource:0}: Error finding container adcfa710c231805411977aaeba54bf1f2078cf927711f2921132b77dd7b0b5b6: Status 404 returned error can't find the container with id adcfa710c231805411977aaeba54bf1f2078cf927711f2921132b77dd7b0b5b6 Jul 24 13:39:47.213575 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.213542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:47.278471 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.278431 2581 generic.go:356] "Generic (PLEG): container finished" podID="0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51" containerID="7dd1ba66c63c31c3c39821f48cebd4b08201ee8e64c52d87b6768f22c846ce0d" exitCode=0 Jul 24 13:39:47.278681 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.278512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vrwt7" event={"ID":"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51","Type":"ContainerDied","Data":"7dd1ba66c63c31c3c39821f48cebd4b08201ee8e64c52d87b6768f22c846ce0d"} Jul 24 13:39:47.279675 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.279643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqkcf" event={"ID":"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e","Type":"ContainerStarted","Data":"adcfa710c231805411977aaeba54bf1f2078cf927711f2921132b77dd7b0b5b6"} Jul 24 13:39:47.336814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:47.336783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:48.094888 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.094749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:48.186680 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.186639 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:48.278279 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.278238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:48.286818 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.286782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vrwt7" event={"ID":"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51","Type":"ContainerStarted","Data":"49b0dedbbbd616132950127e153615fc5229fb9a8657f00672b6c7a3005465d4"} Jul 24 13:39:48.286981 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.286846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vrwt7" event={"ID":"0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51","Type":"ContainerStarted","Data":"f3413ae69394d724ca3f46b55998c7ae3d92d8472866c057a046077f01fe16dc"} Jul 24 13:39:48.314470 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.314404 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vrwt7" podStartSLOduration=2.265303821 podStartE2EDuration="3.314386237s" totalImagesPullingTime="980.572413ms" totalInitContainerRuntime="68.510003ms" isStatefulPod=true podCreationTimestamp="2026-07-24 13:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:48.312970989 +0000 UTC m=+81.909535055" watchObservedRunningTime="2026-07-24 13:39:48.314386237 +0000 UTC m=+81.910950288" Jul 24 13:39:48.459042 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:48.458916 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:49.871245 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:49.871207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:49.951034 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:49.950988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:50.309455 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:50.309405 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:50.415161 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:50.415131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:50.510537 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:50.510502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:51.393627 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:51.393577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:51.890368 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:51.890310 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:51.968502 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:51.968462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:52.303902 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:52.303867 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqkcf" event={"ID":"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e","Type":"ContainerStarted","Data":"cb27a3bf7a7b5e66de148ad49510d038bc2035553f2393e20238e3d1403b7376"} Jul 24 13:39:52.496546 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:52.496510 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:52.597238 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:52.597207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:53.676039 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:53.676005 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:53.775254 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:53.775207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:55.408358 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:55.408323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:55.508938 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:55.508906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:56.323436 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:56.323403 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqkcf" event={"ID":"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e","Type":"ContainerStarted","Data":"422ed46a65574cca410ade9334ad920ae4c47c48796139ad7fe39f747c7a9ad6"} Jul 24 13:39:56.678289 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:56.678247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:56.780697 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:56.780669 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:57.428598 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:57.428556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:39:57.497146 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:39:57.497103 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:00.339545 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:00.339502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cqkcf" event={"ID":"3834f8c9-b9c9-4128-bd5e-cfd2a694e56e","Type":"ContainerStarted","Data":"add0691a83cc0de00797bb09624129c9254fff8fb30460c7b55dab3ee06f1c2a"} Jul 24 13:40:00.359243 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:00.359095 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cqkcf" podStartSLOduration=9.233348266 podStartE2EDuration="21.359078342s" totalImagesPullingTime="12.125730076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:39:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:00.358716874 +0000 UTC m=+93.955280928" watchObservedRunningTime="2026-07-24 13:40:00.359078342 +0000 UTC m=+93.955642395" Jul 24 13:40:02.208664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:02.208626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:02.305027 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:02.304990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:03.589625 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:03.589574 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:04.120527 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:04.120493 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:04.186849 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:04.186820 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:04.398316 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:04.398221 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:04.499021 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:04.498989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:10.683580 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:10.683547 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:11.247271 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:11.247238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:11.253062 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:11.253041 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mts6c" Jul 24 13:40:11.349012 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:11.348981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:12.366164 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:12.366129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:12.426858 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:12.426820 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:12.710268 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:12.710226 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:12.811221 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:12.811185 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:13.497739 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:13.497712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:32.203139 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:32.203100 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:32.280048 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:32.280018 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:39.034786 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:39.034755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:39.135749 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:39.135715 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:39.200860 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:39.200826 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:48.677138 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:48.677089 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:52.629167 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:52.629131 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:40:52.703579 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:40:52.703541 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:04.360831 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:04.360795 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:04.635994 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:04.635912 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:09.290910 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:09.290874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:09.361974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:09.361939 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:13.149781 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:13.149749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:13.213847 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:13.213819 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:13.300054 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:13.300028 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:23.393451 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:23.393355 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:23.460718 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:23.460681 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:52.166117 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:52.166082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:41:52.264271 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:41:52.264236 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:19.603537 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:19.603504 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:19.704309 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:19.704273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:22.160425 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:22.160391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:22.227659 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:22.227629 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:31.588038 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:31.588006 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:31.654305 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:31.654269 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:33.312667 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:33.312632 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:33.414090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:33.414056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:33.477536 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:33.477506 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:47.211794 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:47.211763 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:47.313100 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:47.313065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:42:47.920432 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:42:47.920392 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:26.789488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:26.789432 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 13:43:32.652616 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:32.652553 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:34.309566 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:34.309529 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:54.366566 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:54.366529 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:55.135690 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:55.135654 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:58.000441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:58.000393 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:58.123780 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:58.123744 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:43:59.970798 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:43:59.970767 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:00.162294 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:00.162258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:02.337579 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:02.337546 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:02.729457 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:02.729426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:04.980220 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:04.980179 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:05.081458 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:05.081428 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:10.512461 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:10.512406 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:11.360834 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.360799 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:11.396768 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.396732 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-97vdk"] Jul 24 13:44:11.399893 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.399873 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.402403 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.402373 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 13:44:11.402403 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.402377 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-sjntw\"" Jul 24 13:44:11.403267 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.403243 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 13:44:11.410135 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.410112 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-97vdk"] Jul 24 13:44:11.461702 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.461668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:11.465843 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.465820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.465933 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.465850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmx8g\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-kube-api-access-jmx8g\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.566688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.566654 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.566688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.566689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmx8g\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-kube-api-access-jmx8g\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.575542 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.575515 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-bound-sa-token\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.575736 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.575663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmx8g\" (UniqueName: \"kubernetes.io/projected/51c73b5a-6337-4b1d-ac26-c0e81e6585f8-kube-api-access-jmx8g\") pod \"cert-manager-55b4d954d5-97vdk\" (UID: \"51c73b5a-6337-4b1d-ac26-c0e81e6585f8\") " pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.709716 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.709676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-97vdk" Jul 24 13:44:11.837686 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.837546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-97vdk"] Jul 24 13:44:11.840265 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:44:11.840238 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c73b5a_6337_4b1d_ac26_c0e81e6585f8.slice/crio-115036e9b8a01fe266b2358d5a690682dde19edd0b6ad11c58b09e2ffaee0701 WatchSource:0}: Error finding container 115036e9b8a01fe266b2358d5a690682dde19edd0b6ad11c58b09e2ffaee0701: Status 404 returned error can't find the container with id 115036e9b8a01fe266b2358d5a690682dde19edd0b6ad11c58b09e2ffaee0701 Jul 24 13:44:11.842212 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:11.842197 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:44:12.159181 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:12.159099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-97vdk" event={"ID":"51c73b5a-6337-4b1d-ac26-c0e81e6585f8","Type":"ContainerStarted","Data":"115036e9b8a01fe266b2358d5a690682dde19edd0b6ad11c58b09e2ffaee0701"} Jul 24 13:44:16.176600 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:16.176559 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-97vdk" event={"ID":"51c73b5a-6337-4b1d-ac26-c0e81e6585f8","Type":"ContainerStarted","Data":"5020b38110a57c5b419601f0152d5b8a2b06373329c7f5ba733ad59cd5ea7c7f"} Jul 24 13:44:16.192719 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:16.192671 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-97vdk" podStartSLOduration=1.864082196 podStartE2EDuration="5.192650048s" totalImagesPullingTime="3.328567852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:16.192009073 +0000 UTC m=+349.788573130" watchObservedRunningTime="2026-07-24 13:44:16.192650048 +0000 UTC m=+349.789214106" Jul 24 13:44:19.073749 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:19.073659 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:24.527944 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:24.527911 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:24.628518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:24.628490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:33.654443 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:33.654409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:34.757259 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:34.757211 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:34.813293 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:34.813258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:38.168475 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:38.168437 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:38.276821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:38.276782 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:38.804602 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:38.804554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:38.971072 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:38.971033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.068032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.067944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.102024 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.101984 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm"] Jul 24 13:44:39.106404 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.106382 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.109038 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.109016 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 13:44:39.109383 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.109366 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 13:44:39.115007 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.114983 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 13:44:39.115214 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.115200 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 13:44:39.115276 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.115226 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q2sql\"" Jul 24 13:44:39.141018 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.140981 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm"] Jul 24 13:44:39.152928 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.152895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.153081 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.152940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.153081 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.152967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.153081 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.153020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.153081 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.153054 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zdzg\" (UniqueName: \"kubernetes.io/projected/cf984aa7-a948-4497-8f92-7abb99aaf835-kube-api-access-7zdzg\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.253708 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.253677 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.253708 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.253710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.254166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.253733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.254166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.253771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.254166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.253819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zdzg\" (UniqueName: \"kubernetes.io/projected/cf984aa7-a948-4497-8f92-7abb99aaf835-kube-api-access-7zdzg\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.254166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.254144 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.256402 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.256371 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf984aa7-a948-4497-8f92-7abb99aaf835-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.256688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.256670 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.256688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.256679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf984aa7-a948-4497-8f92-7abb99aaf835-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.262453 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.262415 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zdzg\" (UniqueName: \"kubernetes.io/projected/cf984aa7-a948-4497-8f92-7abb99aaf835-kube-api-access-7zdzg\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-vw6hm\" (UID: \"cf984aa7-a948-4497-8f92-7abb99aaf835\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.315766 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.315726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.416027 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.415998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:39.417039 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.417017 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.520527 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.520493 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.549790 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.549760 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm"] Jul 24 13:44:39.553600 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:44:39.553564 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf984aa7_a948_4497_8f92_7abb99aaf835.slice/crio-7ce8e4260e4127b7b26aea47280e5d426c72dd7a31362e776d8d73b94fdf2c0f WatchSource:0}: Error finding container 7ce8e4260e4127b7b26aea47280e5d426c72dd7a31362e776d8d73b94fdf2c0f: Status 404 returned error can't find the container with id 7ce8e4260e4127b7b26aea47280e5d426c72dd7a31362e776d8d73b94fdf2c0f Jul 24 13:44:39.589057 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.589022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.658228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.658194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:39.760951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:39.760868 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:40.258334 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:40.258271 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" event={"ID":"cf984aa7-a948-4497-8f92-7abb99aaf835","Type":"ContainerStarted","Data":"7ce8e4260e4127b7b26aea47280e5d426c72dd7a31362e776d8d73b94fdf2c0f"} Jul 24 13:44:41.358009 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:41.357969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:41.474816 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:41.474783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:43.270318 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:43.270280 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" event={"ID":"cf984aa7-a948-4497-8f92-7abb99aaf835","Type":"ContainerStarted","Data":"a23d017f45808e0a8dfa6c962595f72e456ef74c7c3f203f3abbee0ac010facf"} Jul 24 13:44:44.275008 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:44.274972 2581 generic.go:356] "Generic (PLEG): container finished" podID="cf984aa7-a948-4497-8f92-7abb99aaf835" containerID="a23d017f45808e0a8dfa6c962595f72e456ef74c7c3f203f3abbee0ac010facf" exitCode=0 Jul 24 13:44:44.275452 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:44.275059 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" event={"ID":"cf984aa7-a948-4497-8f92-7abb99aaf835","Type":"ContainerDied","Data":"a23d017f45808e0a8dfa6c962595f72e456ef74c7c3f203f3abbee0ac010facf"} Jul 24 13:44:45.280252 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:45.280214 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" event={"ID":"cf984aa7-a948-4497-8f92-7abb99aaf835","Type":"ContainerStarted","Data":"3fe4ef8413b6311627a158bfdf9e4c97964a72db25bdbd7c5621eb031d6e0cba"} Jul 24 13:44:45.280762 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:45.280335 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:45.303924 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:45.303872 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" podStartSLOduration=1.998751859 podStartE2EDuration="6.303854737s" totalImagesPullingTime="3.258455765s" totalInitContainerRuntime="1.046647113s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:45.30319766 +0000 UTC m=+378.899761715" watchObservedRunningTime="2026-07-24 13:44:45.303854737 +0000 UTC m=+378.900418792" Jul 24 13:44:51.303823 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:51.303786 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:51.358693 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:51.358656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:51.735223 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:51.735193 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:56.286755 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:56.286725 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-vw6hm" Jul 24 13:44:57.886071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:57.886038 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.009026 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.008989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.481827 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.481796 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.510272 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.510235 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg"] Jul 24 13:44:58.513550 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.513523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.516026 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.516001 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 13:44:58.516210 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.516191 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 13:44:58.516784 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.516767 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xf7q8\"" Jul 24 13:44:58.516855 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.516803 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 13:44:58.516913 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.516803 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 13:44:58.527064 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.527037 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg"] Jul 24 13:44:58.596515 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.596472 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9c5tp"] Jul 24 13:44:58.598911 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.598885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809238b9-66fe-4c31-9763-f1bc2aa459f1-tmp\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.599015 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.598922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwn2\" (UniqueName: \"kubernetes.io/projected/809238b9-66fe-4c31-9763-f1bc2aa459f1-kube-api-access-vwwn2\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.599015 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.598983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.599996 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.599977 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.602215 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.602189 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 13:44:58.602367 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.602219 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b2dcr\"" Jul 24 13:44:58.609354 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.609315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9c5tp"] Jul 24 13:44:58.635272 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.635235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.699991 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.699947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.700010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4cp\" (UniqueName: \"kubernetes.io/projected/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-kube-api-access-ff4cp\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.700060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809238b9-66fe-4c31-9763-f1bc2aa459f1-tmp\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.700088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:44:58.700102 2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.700132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwn2\" (UniqueName: \"kubernetes.io/projected/809238b9-66fe-4c31-9763-f1bc2aa459f1-kube-api-access-vwwn2\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.700213 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:44:58.700190 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs podName:809238b9-66fe-4c31-9763-f1bc2aa459f1 nodeName:}" failed. No retries permitted until 2026-07-24 13:44:59.200165883 +0000 UTC m=+392.796729916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs") pod "kube-auth-proxy-7f8885cbd9-fmfxg" (UID: "809238b9-66fe-4c31-9763-f1bc2aa459f1") : secret "kube-auth-proxy-tls" not found Jul 24 13:44:58.702558 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.702522 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/809238b9-66fe-4c31-9763-f1bc2aa459f1-tmp\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.710175 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.710141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwn2\" (UniqueName: \"kubernetes.io/projected/809238b9-66fe-4c31-9763-f1bc2aa459f1-kube-api-access-vwwn2\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:58.733193 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.733105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.801470 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.801435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.801686 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.801545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4cp\" (UniqueName: \"kubernetes.io/projected/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-kube-api-access-ff4cp\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.801686 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:44:58.801605 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 13:44:58.801797 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:44:58.801690 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert podName:8c1e1f82-e037-40d2-92a8-cefff59ae3e1 nodeName:}" failed. No retries permitted until 2026-07-24 13:44:59.3016706 +0000 UTC m=+392.898234641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert") pod "odh-model-controller-7998c67dc4-9c5tp" (UID: "8c1e1f82-e037-40d2-92a8-cefff59ae3e1") : secret "odh-model-controller-webhook-cert" not found Jul 24 13:44:58.810022 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.809993 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4cp\" (UniqueName: \"kubernetes.io/projected/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-kube-api-access-ff4cp\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:58.830114 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.830086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:58.930641 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:58.930599 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.030525 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.030436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.204473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.204437 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:59.206883 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.206859 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/809238b9-66fe-4c31-9763-f1bc2aa459f1-tls-certs\") pod \"kube-auth-proxy-7f8885cbd9-fmfxg\" (UID: \"809238b9-66fe-4c31-9763-f1bc2aa459f1\") " pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:59.305803 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.305707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:59.308249 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.308208 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8c1e1f82-e037-40d2-92a8-cefff59ae3e1-cert\") pod \"odh-model-controller-7998c67dc4-9c5tp\" (UID: \"8c1e1f82-e037-40d2-92a8-cefff59ae3e1\") " pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:59.424477 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.424444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" Jul 24 13:44:59.487140 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.487094 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.512020 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.511761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:44:59.573964 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.573914 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg"] Jul 24 13:44:59.666843 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.666802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9c5tp"] Jul 24 13:44:59.669557 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:44:59.669531 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1e1f82_e037_40d2_92a8_cefff59ae3e1.slice/crio-db9c182ae5c5d7bfe24f6daa836423fe4d7b3f6645ea73f15c78704e60c02e97 WatchSource:0}: Error finding container db9c182ae5c5d7bfe24f6daa836423fe4d7b3f6645ea73f15c78704e60c02e97: Status 404 returned error can't find the container with id db9c182ae5c5d7bfe24f6daa836423fe4d7b3f6645ea73f15c78704e60c02e97 Jul 24 13:44:59.727597 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.727550 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.733614 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.733567 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:44:59.737829 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.737810 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.739945 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.739927 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 13:44:59.740078 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.740061 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kdb7l\"" Jul 24 13:44:59.740141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.740083 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 13:44:59.740141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.740095 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 13:44:59.740141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.740116 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 13:44:59.750410 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.750378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:44:59.810974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.810942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.810974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.810977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.811230 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.810996 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.811230 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.811019 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.811230 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.811089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.811230 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.811107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.811230 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.811144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8h7\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.825509 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.825425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.897292 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.897262 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:44:59.912488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.912934 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.912705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8h7\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.913263 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.913238 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.915002 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.914978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.915147 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.915055 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.915424 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.915397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.915501 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.915482 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.920228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.920203 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:44:59.920332 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:44:59.920286 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8h7\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7\") pod \"istiod-openshift-gateway-7b7995d5cd-x28b6\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:00.048753 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.048701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:00.191821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.191791 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:45:00.194569 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:45:00.194541 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e763fa0_b626_4d50_bdeb_32dd82e7acf7.slice/crio-ac9fa16c1e19757fd53e8bdf0a6b8c42e4b686a1aba5e3e6f695f72b047da8d3 WatchSource:0}: Error finding container ac9fa16c1e19757fd53e8bdf0a6b8c42e4b686a1aba5e3e6f695f72b047da8d3: Status 404 returned error can't find the container with id ac9fa16c1e19757fd53e8bdf0a6b8c42e4b686a1aba5e3e6f695f72b047da8d3 Jul 24 13:45:00.337439 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.337400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" event={"ID":"0e763fa0-b626-4d50-bdeb-32dd82e7acf7","Type":"ContainerStarted","Data":"ac9fa16c1e19757fd53e8bdf0a6b8c42e4b686a1aba5e3e6f695f72b047da8d3"} Jul 24 13:45:00.339617 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.339571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" event={"ID":"8c1e1f82-e037-40d2-92a8-cefff59ae3e1","Type":"ContainerStarted","Data":"db9c182ae5c5d7bfe24f6daa836423fe4d7b3f6645ea73f15c78704e60c02e97"} Jul 24 13:45:00.341391 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.341366 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" event={"ID":"809238b9-66fe-4c31-9763-f1bc2aa459f1","Type":"ContainerStarted","Data":"4d08fdc35e8c7ca907633bf9cc6a058d642e5eb304b5e9654f13eb2bc43d204b"} Jul 24 13:45:00.739121 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:00.738960 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:04.360921 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:04.360879 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" event={"ID":"8c1e1f82-e037-40d2-92a8-cefff59ae3e1","Type":"ContainerStarted","Data":"69dbb10361b4e1ba5a0070bd271e4838f896c2960dcb3f54f6e0d94de43145f4"} Jul 24 13:45:04.361384 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:04.360977 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:45:04.369659 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:04.369620 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 13:45:04.369795 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:04.369700 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 13:45:04.377978 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:04.377890 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" podStartSLOduration=1.753253169 podStartE2EDuration="6.377867954s" totalImagesPullingTime="4.624614785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:04.37683459 +0000 UTC m=+397.973398648" watchObservedRunningTime="2026-07-24 13:45:04.377867954 +0000 UTC m=+397.974432010" Jul 24 13:45:05.191368 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.191323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:05.292178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.292142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:05.367161 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.367069 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" event={"ID":"0e763fa0-b626-4d50-bdeb-32dd82e7acf7","Type":"ContainerStarted","Data":"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933"} Jul 24 13:45:05.367678 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.367387 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:05.369164 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.369125 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x28b6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 13:45:05.369286 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.369185 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 13:45:05.369286 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.369216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" event={"ID":"809238b9-66fe-4c31-9763-f1bc2aa459f1","Type":"ContainerStarted","Data":"86363d847cf846b2439fd90b51eb254c95bc296b1a6a6f08508a837b17ef33a7"} Jul 24 13:45:05.397022 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.396961 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" podStartSLOduration=2.225104335 podStartE2EDuration="6.39693956s" totalImagesPullingTime="4.171835225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:05.395743016 +0000 UTC m=+398.992307072" watchObservedRunningTime="2026-07-24 13:45:05.39693956 +0000 UTC m=+398.993503614" Jul 24 13:45:05.431074 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.431025 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f8885cbd9-fmfxg" podStartSLOduration=2.712261638 podStartE2EDuration="7.43101007s" totalImagesPullingTime="4.718748432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:05.430379015 +0000 UTC m=+399.026943070" watchObservedRunningTime="2026-07-24 13:45:05.43101007 +0000 UTC m=+399.027574125" Jul 24 13:45:05.597654 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.597620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:05.736377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.736341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:05.933000 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:05.932967 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:06.373289 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:06.373260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:06.750900 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:06.750863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:06.851267 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:06.851235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:09.002313 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.002276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:09.057278 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.057240 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8"] Jul 24 13:45:09.060678 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.060655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.063227 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.063200 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 13:45:09.063363 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.063205 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p9tk5\"" Jul 24 13:45:09.073630 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.073087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8"] Jul 24 13:45:09.080545 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.080519 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc"] Jul 24 13:45:09.081023 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.080999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:09.083680 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.083653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.086025 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.086003 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gqrm9\"" Jul 24 13:45:09.086159 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.086041 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 13:45:09.093776 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.093749 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc"] Jul 24 13:45:09.157982 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.157947 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:09.189507 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.189468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1a1723-1ce9-486e-8e01-e75093a98870-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.189507 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.189503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-cert\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.189745 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.189541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hs5t\" (UniqueName: \"kubernetes.io/projected/6f1a1723-1ce9-486e-8e01-e75093a98870-kube-api-access-9hs5t\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.189745 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.189561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkrx\" (UniqueName: \"kubernetes.io/projected/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-kube-api-access-ktkrx\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.222025 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.221999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:09.290518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.290414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1a1723-1ce9-486e-8e01-e75093a98870-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.290518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.290456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-cert\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.290518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.290504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hs5t\" (UniqueName: \"kubernetes.io/projected/6f1a1723-1ce9-486e-8e01-e75093a98870-kube-api-access-9hs5t\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.290837 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.290535 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkrx\" (UniqueName: \"kubernetes.io/projected/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-kube-api-access-ktkrx\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.293085 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.293061 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-cert\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.293156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.293092 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f1a1723-1ce9-486e-8e01-e75093a98870-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.307306 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.307273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkrx\" (UniqueName: \"kubernetes.io/projected/cbc031e5-28d9-4b1b-876e-ab1bd4e7b946-kube-api-access-ktkrx\") pod \"kserve-controller-manager-d6f4b65b8-4hhd8\" (UID: \"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.307444 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.307306 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hs5t\" (UniqueName: \"kubernetes.io/projected/6f1a1723-1ce9-486e-8e01-e75093a98870-kube-api-access-9hs5t\") pod \"llmisvc-controller-manager-777c6bd7d5-h2jsc\" (UID: \"6f1a1723-1ce9-486e-8e01-e75093a98870\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.375615 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.375558 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:09.395764 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.395732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:09.530805 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.530746 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8"] Jul 24 13:45:09.533794 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:45:09.533765 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc031e5_28d9_4b1b_876e_ab1bd4e7b946.slice/crio-1f77ee711c2a0ba01f1a45c3e922de453dbc79b784586e1f40159f4167337a9e WatchSource:0}: Error finding container 1f77ee711c2a0ba01f1a45c3e922de453dbc79b784586e1f40159f4167337a9e: Status 404 returned error can't find the container with id 1f77ee711c2a0ba01f1a45c3e922de453dbc79b784586e1f40159f4167337a9e Jul 24 13:45:09.550833 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:09.550771 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc"] Jul 24 13:45:09.553264 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:45:09.553233 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6f1a1723_1ce9_486e_8e01_e75093a98870.slice/crio-72ebc7a789ea232e339a83b0160e109764b4532e365d7ce990091fd267dbd06a WatchSource:0}: Error finding container 72ebc7a789ea232e339a83b0160e109764b4532e365d7ce990091fd267dbd06a: Status 404 returned error can't find the container with id 72ebc7a789ea232e339a83b0160e109764b4532e365d7ce990091fd267dbd06a Jul 24 13:45:10.312445 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:10.312374 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:10.390871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:10.390836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" event={"ID":"6f1a1723-1ce9-486e-8e01-e75093a98870","Type":"ContainerStarted","Data":"72ebc7a789ea232e339a83b0160e109764b4532e365d7ce990091fd267dbd06a"} Jul 24 13:45:10.392154 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:10.392124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" event={"ID":"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946","Type":"ContainerStarted","Data":"1f77ee711c2a0ba01f1a45c3e922de453dbc79b784586e1f40159f4167337a9e"} Jul 24 13:45:10.403064 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:10.403021 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:10.500695 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:10.500656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:13.407352 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.407319 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" event={"ID":"cbc031e5-28d9-4b1b-876e-ab1bd4e7b946","Type":"ContainerStarted","Data":"1d6997d082fa61ff52d8124ece6ffa5f69c2ef784bc3d9b5c45d2b48445208d8"} Jul 24 13:45:13.407808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.407534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:13.408896 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.408873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" event={"ID":"6f1a1723-1ce9-486e-8e01-e75093a98870","Type":"ContainerStarted","Data":"2930e0c35d2d99da1205117edf760238a222eae7154de4a50d20c06a702ee6c0"} Jul 24 13:45:13.409015 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.408999 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:13.427807 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.427735 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" podStartSLOduration=1.590203549 podStartE2EDuration="4.427710714s" totalImagesPullingTime="2.837507165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:13.424932704 +0000 UTC m=+407.021496787" watchObservedRunningTime="2026-07-24 13:45:13.427710714 +0000 UTC m=+407.024274770" Jul 24 13:45:13.445522 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:13.445461 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" podStartSLOduration=1.624812517 podStartE2EDuration="4.445443042s" totalImagesPullingTime="2.820630525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:13.444376454 +0000 UTC m=+407.040940510" watchObservedRunningTime="2026-07-24 13:45:13.445443042 +0000 UTC m=+407.042007097" Jul 24 13:45:14.013786 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.013746 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:45:14.014055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.014015 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerName="discovery" containerID="cri-o://f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933" gracePeriod=30 Jul 24 13:45:14.132036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.131986 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:14.195752 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.195716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:14.263713 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.263687 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:14.332327 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332235 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332327 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332327 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332315 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332645 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332333 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332645 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332360 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc8h7\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332645 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332421 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332645 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332446 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token\") pod \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\" (UID: \"0e763fa0-b626-4d50-bdeb-32dd82e7acf7\") " Jul 24 13:45:14.332867 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.332799 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:45:14.334816 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.334763 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:45:14.334946 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.334868 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:45:14.334946 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.334926 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts" (OuterVolumeSpecName: "cacerts") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:45:14.335043 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.334931 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7" (OuterVolumeSpecName: "kube-api-access-mc8h7") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "kube-api-access-mc8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:45:14.335043 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.334981 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs" (OuterVolumeSpecName: "local-certs") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:45:14.335043 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.335013 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e763fa0-b626-4d50-bdeb-32dd82e7acf7" (UID: "0e763fa0-b626-4d50-bdeb-32dd82e7acf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:45:14.413918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.413885 2581 generic.go:356] "Generic (PLEG): container finished" podID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerID="f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933" exitCode=0 Jul 24 13:45:14.414320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.413929 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" Jul 24 13:45:14.414320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.413954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" event={"ID":"0e763fa0-b626-4d50-bdeb-32dd82e7acf7","Type":"ContainerDied","Data":"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933"} Jul 24 13:45:14.414320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.414002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6" event={"ID":"0e763fa0-b626-4d50-bdeb-32dd82e7acf7","Type":"ContainerDied","Data":"ac9fa16c1e19757fd53e8bdf0a6b8c42e4b686a1aba5e3e6f695f72b047da8d3"} Jul 24 13:45:14.414320 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.414022 2581 scope.go:117] "RemoveContainer" containerID="f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933" Jul 24 13:45:14.423090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.423067 2581 scope.go:117] "RemoveContainer" containerID="f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933" Jul 24 13:45:14.423398 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:45:14.423377 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933\": container with ID starting with f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933 not found: ID does not exist" containerID="f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933" Jul 24 13:45:14.423462 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.423409 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933"} err="failed to get container status \"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933\": rpc error: code = NotFound desc = could not find container \"f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933\": container with ID starting with f1d2be6bb635ebf24c856a211e49ee146df54bcd4975fafc01d1eaca23123933 not found: ID does not exist" Jul 24 13:45:14.433656 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433629 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-ca-configmap\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433656 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433657 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-local-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433672 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-cacerts\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433682 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8h7\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-kube-api-access-mc8h7\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433690 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-csr-dns-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433698 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.433856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.433706 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e763fa0-b626-4d50-bdeb-32dd82e7acf7-istio-kubeconfig\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:45:14.435488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.435467 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:45:14.440191 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.440166 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x28b6"] Jul 24 13:45:14.915929 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:14.915897 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" path="/var/lib/kubelet/pods/0e763fa0-b626-4d50-bdeb-32dd82e7acf7/volumes" Jul 24 13:45:15.372298 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:15.372215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9c5tp" Jul 24 13:45:38.989820 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:38.989787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:39.049971 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:39.049934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:44.416577 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:44.416544 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h2jsc" Jul 24 13:45:44.419404 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:44.419384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4hhd8" Jul 24 13:45:50.720939 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:50.720861 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:50.826488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:50.826454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:59.446376 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.446336 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:59.545441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.545405 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:59.589934 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.589902 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:45:59.590208 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.590196 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerName="discovery" Jul 24 13:45:59.590261 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.590209 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerName="discovery" Jul 24 13:45:59.590261 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.590258 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e763fa0-b626-4d50-bdeb-32dd82e7acf7" containerName="discovery" Jul 24 13:45:59.592787 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.592765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:45:59.595276 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.595253 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 13:45:59.595891 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.595872 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-29zvt\"" Jul 24 13:45:59.595968 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.595876 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 13:45:59.603755 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.603729 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:45:59.635624 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.635573 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:45:59.671024 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.670989 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhm5\" (UniqueName: \"kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5\") pod \"kuadrant-operator-catalog-gb24v\" (UID: \"3b082396-c51f-478d-9922-3df2f77f97b8\") " pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:45:59.772205 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.772097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhm5\" (UniqueName: \"kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5\") pod \"kuadrant-operator-catalog-gb24v\" (UID: \"3b082396-c51f-478d-9922-3df2f77f97b8\") " pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:45:59.783610 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.783568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhm5\" (UniqueName: \"kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5\") pod \"kuadrant-operator-catalog-gb24v\" (UID: \"3b082396-c51f-478d-9922-3df2f77f97b8\") " pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:45:59.903599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.903540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:45:59.962259 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:45:59.962228 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:46:00.035651 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.035576 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:46:00.038371 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:00.038343 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b082396_c51f_478d_9922_3df2f77f97b8.slice/crio-1b20e7dd6b90e8ed51e9bac266d2f1db3bea57098d3e9392bad18aae659fa14c WatchSource:0}: Error finding container 1b20e7dd6b90e8ed51e9bac266d2f1db3bea57098d3e9392bad18aae659fa14c: Status 404 returned error can't find the container with id 1b20e7dd6b90e8ed51e9bac266d2f1db3bea57098d3e9392bad18aae659fa14c Jul 24 13:46:00.172937 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.172904 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c7vzn"] Jul 24 13:46:00.177141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.177121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:00.185103 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.185078 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c7vzn"] Jul 24 13:46:00.275227 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.275193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd6k\" (UniqueName: \"kubernetes.io/projected/894b201b-d82e-48be-9c3d-f46648c37034-kube-api-access-hpd6k\") pod \"kuadrant-operator-catalog-c7vzn\" (UID: \"894b201b-d82e-48be-9c3d-f46648c37034\") " pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:00.375946 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.375826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd6k\" (UniqueName: \"kubernetes.io/projected/894b201b-d82e-48be-9c3d-f46648c37034-kube-api-access-hpd6k\") pod \"kuadrant-operator-catalog-c7vzn\" (UID: \"894b201b-d82e-48be-9c3d-f46648c37034\") " pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:00.384406 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.384380 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd6k\" (UniqueName: \"kubernetes.io/projected/894b201b-d82e-48be-9c3d-f46648c37034-kube-api-access-hpd6k\") pod \"kuadrant-operator-catalog-c7vzn\" (UID: \"894b201b-d82e-48be-9c3d-f46648c37034\") " pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:00.487668 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.487631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:00.588156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.588107 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" event={"ID":"3b082396-c51f-478d-9922-3df2f77f97b8","Type":"ContainerStarted","Data":"1b20e7dd6b90e8ed51e9bac266d2f1db3bea57098d3e9392bad18aae659fa14c"} Jul 24 13:46:00.628180 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:00.628101 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c7vzn"] Jul 24 13:46:00.645395 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:00.645361 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894b201b_d82e_48be_9c3d_f46648c37034.slice/crio-6b3af452d7011994eef96154d5cd1e08c422eb9af3a81e793f57df8db6734260 WatchSource:0}: Error finding container 6b3af452d7011994eef96154d5cd1e08c422eb9af3a81e793f57df8db6734260: Status 404 returned error can't find the container with id 6b3af452d7011994eef96154d5cd1e08c422eb9af3a81e793f57df8db6734260 Jul 24 13:46:01.594739 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:01.594699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" event={"ID":"894b201b-d82e-48be-9c3d-f46648c37034","Type":"ContainerStarted","Data":"6b3af452d7011994eef96154d5cd1e08c422eb9af3a81e793f57df8db6734260"} Jul 24 13:46:02.599976 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.599934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" event={"ID":"894b201b-d82e-48be-9c3d-f46648c37034","Type":"ContainerStarted","Data":"e16fe175e8b239def20fddd53e3a0b1f09e941fdb5f56e763ab1237d51e30087"} Jul 24 13:46:02.601371 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.601346 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" event={"ID":"3b082396-c51f-478d-9922-3df2f77f97b8","Type":"ContainerStarted","Data":"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d"} Jul 24 13:46:02.601470 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.601411 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" podUID="3b082396-c51f-478d-9922-3df2f77f97b8" containerName="registry-server" containerID="cri-o://6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d" gracePeriod=2 Jul 24 13:46:02.615907 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.615858 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" podStartSLOduration=1.165540408 podStartE2EDuration="2.615842647s" totalImagesPullingTime="1.450302239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:02.615292619 +0000 UTC m=+456.211856675" watchObservedRunningTime="2026-07-24 13:46:02.615842647 +0000 UTC m=+456.212406701" Jul 24 13:46:02.629670 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.629619 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" podStartSLOduration=1.590999832 podStartE2EDuration="3.629580114s" totalImagesPullingTime="2.038580282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:02.629190682 +0000 UTC m=+456.225754749" watchObservedRunningTime="2026-07-24 13:46:02.629580114 +0000 UTC m=+456.226144231" Jul 24 13:46:02.851510 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.851435 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:46:02.898232 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.898195 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwhm5\" (UniqueName: \"kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5\") pod \"3b082396-c51f-478d-9922-3df2f77f97b8\" (UID: \"3b082396-c51f-478d-9922-3df2f77f97b8\") " Jul 24 13:46:02.900420 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.900385 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5" (OuterVolumeSpecName: "kube-api-access-pwhm5") pod "3b082396-c51f-478d-9922-3df2f77f97b8" (UID: "3b082396-c51f-478d-9922-3df2f77f97b8"). InnerVolumeSpecName "kube-api-access-pwhm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:46:02.999637 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:02.999573 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhm5\" (UniqueName: \"kubernetes.io/projected/3b082396-c51f-478d-9922-3df2f77f97b8-kube-api-access-pwhm5\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:46:03.606516 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.606478 2581 generic.go:356] "Generic (PLEG): container finished" podID="3b082396-c51f-478d-9922-3df2f77f97b8" containerID="6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d" exitCode=0 Jul 24 13:46:03.607016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.606522 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" Jul 24 13:46:03.607016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.606568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" event={"ID":"3b082396-c51f-478d-9922-3df2f77f97b8","Type":"ContainerDied","Data":"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d"} Jul 24 13:46:03.607016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.606636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gb24v" event={"ID":"3b082396-c51f-478d-9922-3df2f77f97b8","Type":"ContainerDied","Data":"1b20e7dd6b90e8ed51e9bac266d2f1db3bea57098d3e9392bad18aae659fa14c"} Jul 24 13:46:03.607016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.606661 2581 scope.go:117] "RemoveContainer" containerID="6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d" Jul 24 13:46:03.615066 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.615044 2581 scope.go:117] "RemoveContainer" containerID="6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d" Jul 24 13:46:03.615397 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:46:03.615368 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d\": container with ID starting with 6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d not found: ID does not exist" containerID="6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d" Jul 24 13:46:03.615475 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.615406 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d"} err="failed to get container status \"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d\": rpc error: code = NotFound desc = could not find container \"6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d\": container with ID starting with 6cd7137105220c2db5328d3890fb3354bc492609e1893f50fe4bfde2f6ddcf5d not found: ID does not exist" Jul 24 13:46:03.621238 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.621206 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:46:03.625109 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:03.625077 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gb24v"] Jul 24 13:46:04.915726 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:04.915692 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b082396-c51f-478d-9922-3df2f77f97b8" path="/var/lib/kubelet/pods/3b082396-c51f-478d-9922-3df2f77f97b8/volumes" Jul 24 13:46:10.488244 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:10.488205 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:10.488795 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:10.488273 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:10.509636 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:10.509577 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:10.655346 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:10.655321 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-c7vzn" Jul 24 13:46:28.213676 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:28.213646 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:28.805819 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:28.805781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:32.054619 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.054565 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4lffm"] Jul 24 13:46:32.055133 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.054970 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b082396-c51f-478d-9922-3df2f77f97b8" containerName="registry-server" Jul 24 13:46:32.055133 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.054988 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b082396-c51f-478d-9922-3df2f77f97b8" containerName="registry-server" Jul 24 13:46:32.055133 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.055067 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b082396-c51f-478d-9922-3df2f77f97b8" containerName="registry-server" Jul 24 13:46:32.061262 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.061236 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:32.063417 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.063394 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-th6rc\"" Jul 24 13:46:32.066966 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.066945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4lffm"] Jul 24 13:46:32.120673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.120636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28hj7\" (UniqueName: \"kubernetes.io/projected/f39c92bf-6319-4b13-ab01-c06aba5a8fdc-kube-api-access-28hj7\") pod \"authorino-operator-75cc8d5cb7-4lffm\" (UID: \"f39c92bf-6319-4b13-ab01-c06aba5a8fdc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:32.221987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.221954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28hj7\" (UniqueName: \"kubernetes.io/projected/f39c92bf-6319-4b13-ab01-c06aba5a8fdc-kube-api-access-28hj7\") pod \"authorino-operator-75cc8d5cb7-4lffm\" (UID: \"f39c92bf-6319-4b13-ab01-c06aba5a8fdc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:32.230531 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.230491 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28hj7\" (UniqueName: \"kubernetes.io/projected/f39c92bf-6319-4b13-ab01-c06aba5a8fdc-kube-api-access-28hj7\") pod \"authorino-operator-75cc8d5cb7-4lffm\" (UID: \"f39c92bf-6319-4b13-ab01-c06aba5a8fdc\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:32.371660 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.371548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:32.507462 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.507438 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-4lffm"] Jul 24 13:46:32.510120 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:32.510093 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39c92bf_6319_4b13_ab01_c06aba5a8fdc.slice/crio-3cc3260f8dac77567d41d1c2ed44c8c78530c48a1a517e67db9ab33e4088d528 WatchSource:0}: Error finding container 3cc3260f8dac77567d41d1c2ed44c8c78530c48a1a517e67db9ab33e4088d528: Status 404 returned error can't find the container with id 3cc3260f8dac77567d41d1c2ed44c8c78530c48a1a517e67db9ab33e4088d528 Jul 24 13:46:32.719433 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:32.719396 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" event={"ID":"f39c92bf-6319-4b13-ab01-c06aba5a8fdc","Type":"ContainerStarted","Data":"3cc3260f8dac77567d41d1c2ed44c8c78530c48a1a517e67db9ab33e4088d528"} Jul 24 13:46:33.466117 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:33.466076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:33.554684 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:33.554646 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:35.336455 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.336416 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl"] Jul 24 13:46:35.339781 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.339755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:35.342343 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.342325 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-r95wk\"" Jul 24 13:46:35.349575 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.349551 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl"] Jul 24 13:46:35.450400 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.450364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfcs\" (UniqueName: \"kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs\") pod \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" (UID: \"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:35.551767 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.551725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfcs\" (UniqueName: \"kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs\") pod \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" (UID: \"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:35.563536 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.563509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfcs\" (UniqueName: \"kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs\") pod \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" (UID: \"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:35.649171 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.649079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:35.649350 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.649288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:35.740628 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.740596 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" event={"ID":"f39c92bf-6319-4b13-ab01-c06aba5a8fdc","Type":"ContainerStarted","Data":"50d42a47ef569402451b4d3b0bce9cc4ba2f823c328b452fa9869109b8dfd4e3"} Jul 24 13:46:35.740805 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.740703 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:35.753885 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.753853 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:35.769165 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.769086 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" podStartSLOduration=1.619614976 podStartE2EDuration="3.769069304s" totalImagesPullingTime="2.149454328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:35.768648304 +0000 UTC m=+489.365212358" watchObservedRunningTime="2026-07-24 13:46:35.769069304 +0000 UTC m=+489.365633359" Jul 24 13:46:35.797758 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:35.797668 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl"] Jul 24 13:46:35.800262 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:35.800234 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe6ae90_2c1b_4b1d_8fbf_ecda51e14f2a.slice/crio-81279c07b97a958636175b5388e799516e730e8f9ab8dd4d498731daf5e1d77c WatchSource:0}: Error finding container 81279c07b97a958636175b5388e799516e730e8f9ab8dd4d498731daf5e1d77c: Status 404 returned error can't find the container with id 81279c07b97a958636175b5388e799516e730e8f9ab8dd4d498731daf5e1d77c Jul 24 13:46:36.422961 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:36.422929 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:36.747249 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:36.747172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" event={"ID":"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a","Type":"ContainerStarted","Data":"81279c07b97a958636175b5388e799516e730e8f9ab8dd4d498731daf5e1d77c"} Jul 24 13:46:37.440204 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:37.440165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:38.668292 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.668257 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w"] Jul 24 13:46:38.671406 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.671385 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:38.673717 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.673696 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qpqdl\"" Jul 24 13:46:38.673808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.673706 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 13:46:38.682781 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.682756 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w"] Jul 24 13:46:38.760697 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.760649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" event={"ID":"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a","Type":"ContainerStarted","Data":"e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033"} Jul 24 13:46:38.760865 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.760721 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:38.776351 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.776300 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" podStartSLOduration=1.752683877 podStartE2EDuration="3.776282889s" totalImagesPullingTime="2.023599012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:38.775910166 +0000 UTC m=+492.372474222" watchObservedRunningTime="2026-07-24 13:46:38.776282889 +0000 UTC m=+492.372847012" Jul 24 13:46:38.780168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.780141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhb77\" (UniqueName: \"kubernetes.io/projected/972d23cb-5bcc-448a-b6ce-cfcc1aee743f-kube-api-access-lhb77\") pod \"dns-operator-controller-manager-864dd78c4c-ddv8w\" (UID: \"972d23cb-5bcc-448a-b6ce-cfcc1aee743f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:38.880679 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.880642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhb77\" (UniqueName: \"kubernetes.io/projected/972d23cb-5bcc-448a-b6ce-cfcc1aee743f-kube-api-access-lhb77\") pod \"dns-operator-controller-manager-864dd78c4c-ddv8w\" (UID: \"972d23cb-5bcc-448a-b6ce-cfcc1aee743f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:38.888603 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.888563 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhb77\" (UniqueName: \"kubernetes.io/projected/972d23cb-5bcc-448a-b6ce-cfcc1aee743f-kube-api-access-lhb77\") pod \"dns-operator-controller-manager-864dd78c4c-ddv8w\" (UID: \"972d23cb-5bcc-448a-b6ce-cfcc1aee743f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:38.981778 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:38.981697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:39.121126 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:39.121100 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w"] Jul 24 13:46:39.123867 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:39.123841 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972d23cb_5bcc_448a_b6ce_cfcc1aee743f.slice/crio-6a879cc27458f751755c67a19acefabbf5b914c847a10af6c42255b99edbb001 WatchSource:0}: Error finding container 6a879cc27458f751755c67a19acefabbf5b914c847a10af6c42255b99edbb001: Status 404 returned error can't find the container with id 6a879cc27458f751755c67a19acefabbf5b914c847a10af6c42255b99edbb001 Jul 24 13:46:39.429121 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:39.429088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:39.766441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:39.766354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" event={"ID":"972d23cb-5bcc-448a-b6ce-cfcc1aee743f","Type":"ContainerStarted","Data":"6a879cc27458f751755c67a19acefabbf5b914c847a10af6c42255b99edbb001"} Jul 24 13:46:40.052529 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:40.052453 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:41.777190 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:41.777155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" event={"ID":"972d23cb-5bcc-448a-b6ce-cfcc1aee743f","Type":"ContainerStarted","Data":"01a785ea3fc55d735bb1c3aefee662bd429b6ce112c82e6b0822b172869fc673"} Jul 24 13:46:41.777700 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:41.777276 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:41.795532 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:41.795482 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" podStartSLOduration=1.909921508 podStartE2EDuration="3.795457587s" totalImagesPullingTime="1.885536079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:41.794318162 +0000 UTC m=+495.390882218" watchObservedRunningTime="2026-07-24 13:46:41.795457587 +0000 UTC m=+495.392021681" Jul 24 13:46:43.267185 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.267154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:43.672358 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.672323 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw"] Jul 24 13:46:43.675663 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.675645 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.678346 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.678314 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fsd7k\"" Jul 24 13:46:43.689130 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.689106 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw"] Jul 24 13:46:43.823291 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.823253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.823485 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.823327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqz2\" (UniqueName: \"kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.923944 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.923839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqz2\" (UniqueName: \"kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.923944 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.923891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.924293 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.924272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.931614 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.931560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqz2\" (UniqueName: \"kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:43.985634 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:43.985553 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:44.125684 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:44.125660 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw"] Jul 24 13:46:44.128209 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:44.128178 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f76919f_62af_4134_8344_03b767239562.slice/crio-033e0ddfe355ea303311f98960521fe68fb201c24b70b205b7f6844bcb5a329e WatchSource:0}: Error finding container 033e0ddfe355ea303311f98960521fe68fb201c24b70b205b7f6844bcb5a329e: Status 404 returned error can't find the container with id 033e0ddfe355ea303311f98960521fe68fb201c24b70b205b7f6844bcb5a329e Jul 24 13:46:44.648875 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:44.648846 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:44.792395 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:44.792354 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" event={"ID":"0f76919f-62af-4134-8344-03b767239562","Type":"ContainerStarted","Data":"033e0ddfe355ea303311f98960521fe68fb201c24b70b205b7f6844bcb5a329e"} Jul 24 13:46:44.793077 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:44.793054 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:44.893625 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:44.893570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:45.239892 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:45.239863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:45.340749 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:45.340711 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:45.622685 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:45.622574 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:45.723655 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:45.723623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:46.749920 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:46.749886 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-4lffm" Jul 24 13:46:49.021061 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.021019 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:49.121492 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.121385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:49.768906 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.768878 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:46:49.816493 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.816453 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" event={"ID":"0f76919f-62af-4134-8344-03b767239562","Type":"ContainerStarted","Data":"49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1"} Jul 24 13:46:49.816687 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.816599 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:46:49.837542 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:49.837493 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" podStartSLOduration=2.1422777379999998 podStartE2EDuration="6.837476806s" totalImagesPullingTime="4.695199068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:49.835311177 +0000 UTC m=+503.431875231" watchObservedRunningTime="2026-07-24 13:46:49.837476806 +0000 UTC m=+503.434040861" Jul 24 13:46:51.795930 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.795893 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw"] Jul 24 13:46:51.798529 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.798506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.800618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.800413 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:51.805951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.800777 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 13:46:51.805951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.800784 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2sl75\"" Jul 24 13:46:51.805951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.800988 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 13:46:51.807743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.807715 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw"] Jul 24 13:46:51.888773 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.888736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.888773 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.888778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.889026 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.888922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdx9h\" (UniqueName: \"kubernetes.io/projected/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-kube-api-access-cdx9h\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.990024 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.989977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdx9h\" (UniqueName: \"kubernetes.io/projected/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-kube-api-access-cdx9h\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.990024 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.990030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.990315 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.990053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.990315 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:46:51.990147 2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 13:46:51.990315 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:46:51.990233 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert podName:f6ac7b88-87b5-426e-b0cd-13eb84e239a0 nodeName:}" failed. No retries permitted until 2026-07-24 13:46:52.490212455 +0000 UTC m=+506.086776490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-5ntqw" (UID: "f6ac7b88-87b5-426e-b0cd-13eb84e239a0") : secret "plugin-serving-cert" not found Jul 24 13:46:51.990757 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.990737 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:51.998149 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:51.998122 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdx9h\" (UniqueName: \"kubernetes.io/projected/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-kube-api-access-cdx9h\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:52.046378 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.046291 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:52.419899 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.419856 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:52.474975 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.474945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:52.493958 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.493919 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:52.496382 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.496358 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6ac7b88-87b5-426e-b0cd-13eb84e239a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5ntqw\" (UID: \"f6ac7b88-87b5-426e-b0cd-13eb84e239a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:52.713177 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.713086 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" Jul 24 13:46:52.784166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.784143 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ddv8w" Jul 24 13:46:52.852599 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:52.852476 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw"] Jul 24 13:46:52.855039 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:46:52.855009 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6ac7b88_87b5_426e_b0cd_13eb84e239a0.slice/crio-8c6d5e50b7d9f197da291edf8fc9860198aa4099ec0e260c1dfedc66e6a94ccd WatchSource:0}: Error finding container 8c6d5e50b7d9f197da291edf8fc9860198aa4099ec0e260c1dfedc66e6a94ccd: Status 404 returned error can't find the container with id 8c6d5e50b7d9f197da291edf8fc9860198aa4099ec0e260c1dfedc66e6a94ccd Jul 24 13:46:53.835951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:53.835919 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" event={"ID":"f6ac7b88-87b5-426e-b0cd-13eb84e239a0","Type":"ContainerStarted","Data":"8c6d5e50b7d9f197da291edf8fc9860198aa4099ec0e260c1dfedc66e6a94ccd"} Jul 24 13:46:56.355367 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:56.355332 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:46:56.455334 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:46:56.455293 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:00.822862 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:00.822833 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:01.709854 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.709820 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w"] Jul 24 13:47:01.731159 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.731124 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w"] Jul 24 13:47:01.731323 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.731258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.876940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.876906 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.877379 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.876965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lk8h\" (UniqueName: \"kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.978416 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.978325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lk8h\" (UniqueName: \"kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.978623 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.978440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.978885 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.978859 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:01.995601 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:01.995553 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lk8h\" (UniqueName: \"kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:02.045489 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.045449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:02.562112 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.562072 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w"] Jul 24 13:47:02.570883 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.570851 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w"] Jul 24 13:47:02.580969 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.580931 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw"] Jul 24 13:47:02.581183 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.581155 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" podUID="0f76919f-62af-4134-8344-03b767239562" containerName="manager" containerID="cri-o://49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1" gracePeriod=2 Jul 24 13:47:02.590697 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.590547 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw"] Jul 24 13:47:02.594055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.593340 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:02.594055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.593799 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f76919f-62af-4134-8344-03b767239562" containerName="manager" Jul 24 13:47:02.594055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.593815 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f76919f-62af-4134-8344-03b767239562" containerName="manager" Jul 24 13:47:02.594055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.593900 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f76919f-62af-4134-8344-03b767239562" containerName="manager" Jul 24 13:47:02.594427 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:02.594355 2581 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0f76919f-62af-4134-8344-03b767239562\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:02.605245 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.605211 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl"] Jul 24 13:47:02.605420 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.605356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.605936 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.605728 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" containerName="manager" containerID="cri-o://e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033" gracePeriod=2 Jul 24 13:47:02.605936 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.605874 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:02.607684 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:02.607642 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0f76919f-62af-4134-8344-03b767239562" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:02.612716 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.612512 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:02.612716 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.612678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.616867 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.615861 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl"] Jul 24 13:47:02.622758 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.622723 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:02.628518 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.628459 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c"] Jul 24 13:47:02.629119 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.629095 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" containerName="manager" Jul 24 13:47:02.629119 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.629121 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" containerName="manager" Jul 24 13:47:02.629315 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.629248 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" containerName="manager" Jul 24 13:47:02.633738 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.633668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:02.637034 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:02.636990 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:02.643789 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.643712 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c"] Jul 24 13:47:02.673863 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:02.673821 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0f76919f-62af-4134-8344-03b767239562" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:02.785666 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.785623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvqn\" (UniqueName: \"kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.785871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.785745 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnx5\" (UniqueName: \"kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.785871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.785785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.785871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.785821 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfb4\" (UniqueName: \"kubernetes.io/projected/4c52dabe-b5cc-4328-b714-9027e4bac2e0-kube-api-access-kbfb4\") pod \"limitador-operator-controller-manager-859d7b78bd-jn88c\" (UID: \"4c52dabe-b5cc-4328-b714-9027e4bac2e0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:02.786010 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.785954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.887447 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvqn\" (UniqueName: \"kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.887447 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnx5\" (UniqueName: \"kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.888032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.888032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfb4\" (UniqueName: \"kubernetes.io/projected/4c52dabe-b5cc-4328-b714-9027e4bac2e0-kube-api-access-kbfb4\") pod \"limitador-operator-controller-manager-859d7b78bd-jn88c\" (UID: \"4c52dabe-b5cc-4328-b714-9027e4bac2e0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:02.888032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.888032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:02.888032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.887966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.901264 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.901227 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfb4\" (UniqueName: \"kubernetes.io/projected/4c52dabe-b5cc-4328-b714-9027e4bac2e0-kube-api-access-kbfb4\") pod \"limitador-operator-controller-manager-859d7b78bd-jn88c\" (UID: \"4c52dabe-b5cc-4328-b714-9027e4bac2e0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:02.901801 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.901772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnx5\" (UniqueName: \"kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ps8n7\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:02.902964 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:02.902943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvqn\" (UniqueName: \"kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dz7kd\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:03.018525 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:03.018281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:03.027498 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:03.027461 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:03.034467 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:03.034434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:04.199938 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:04.199850 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:04.301156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:04.301105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:04.793735 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:04.793691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:04.901406 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:04.901371 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:08.196947 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:08.196903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:08.297809 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:08.297774 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:08.511133 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:08.511052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:10.594035 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:10.593998 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:10.656503 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:10.656462 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:14.513751 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.513030 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:14.515413 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.515392 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:14.515814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.515792 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:14.516959 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.516927 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:14.518607 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.518493 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0f76919f-62af-4134-8344-03b767239562" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:14.519972 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.519943 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:14.522554 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:14.522527 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de03b83_b626_4236_9f5a_62434c87281d.slice/crio-aaf1468e8ba7fec1d715bed5f8eae5edd8a31b02b8632278bd6bcf791e851d28 WatchSource:0}: Error finding container aaf1468e8ba7fec1d715bed5f8eae5edd8a31b02b8632278bd6bcf791e851d28: Status 404 returned error can't find the container with id aaf1468e8ba7fec1d715bed5f8eae5edd8a31b02b8632278bd6bcf791e851d28 Jul 24 13:47:14.538287 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.538255 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:14.542627 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:14.542566 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327a19d2_ed61_4610_b98c_a7c88b6049eb.slice/crio-e8e07245d746a0cdce53fd58bcbee8579a0d7c863d0cf0ce9961516eeb4e6e62 WatchSource:0}: Error finding container e8e07245d746a0cdce53fd58bcbee8579a0d7c863d0cf0ce9961516eeb4e6e62: Status 404 returned error can't find the container with id e8e07245d746a0cdce53fd58bcbee8579a0d7c863d0cf0ce9961516eeb4e6e62 Jul 24 13:47:14.560418 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.560396 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c"] Jul 24 13:47:14.563809 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:14.563781 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c52dabe_b5cc_4328_b714_9027e4bac2e0.slice/crio-45d05b25c44f6036bc9a9fee57c144160f6c6896ad4fbcb933eb4337f7738c1d WatchSource:0}: Error finding container 45d05b25c44f6036bc9a9fee57c144160f6c6896ad4fbcb933eb4337f7738c1d: Status 404 returned error can't find the container with id 45d05b25c44f6036bc9a9fee57c144160f6c6896ad4fbcb933eb4337f7738c1d Jul 24 13:47:14.590168 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.590143 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twqz2\" (UniqueName: \"kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2\") pod \"0f76919f-62af-4134-8344-03b767239562\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " Jul 24 13:47:14.590279 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.590179 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume\") pod \"0f76919f-62af-4134-8344-03b767239562\" (UID: \"0f76919f-62af-4134-8344-03b767239562\") " Jul 24 13:47:14.590279 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.590225 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sfcs\" (UniqueName: \"kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs\") pod \"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a\" (UID: \"0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a\") " Jul 24 13:47:14.590852 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.590820 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f76919f-62af-4134-8344-03b767239562" (UID: "0f76919f-62af-4134-8344-03b767239562"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:47:14.592155 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.592127 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2" (OuterVolumeSpecName: "kube-api-access-twqz2") pod "0f76919f-62af-4134-8344-03b767239562" (UID: "0f76919f-62af-4134-8344-03b767239562"). InnerVolumeSpecName "kube-api-access-twqz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:14.592237 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.592217 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs" (OuterVolumeSpecName: "kube-api-access-8sfcs") pod "0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" (UID: "0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a"). InnerVolumeSpecName "kube-api-access-8sfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:14.690980 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.690941 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twqz2\" (UniqueName: \"kubernetes.io/projected/0f76919f-62af-4134-8344-03b767239562-kube-api-access-twqz2\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:14.690980 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.690971 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f76919f-62af-4134-8344-03b767239562-extensions-socket-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:14.690980 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.690984 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sfcs\" (UniqueName: \"kubernetes.io/projected/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a-kube-api-access-8sfcs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:14.916382 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.916355 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f76919f-62af-4134-8344-03b767239562" path="/var/lib/kubelet/pods/0f76919f-62af-4134-8344-03b767239562/volumes" Jul 24 13:47:14.916689 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.916675 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" path="/var/lib/kubelet/pods/0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a/volumes" Jul 24 13:47:14.943825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.943792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" event={"ID":"f6ac7b88-87b5-426e-b0cd-13eb84e239a0","Type":"ContainerStarted","Data":"6c0bba40ec201e6778885c4729514e2b416d6ddc461481c2ac1eb3dbd0aad90c"} Jul 24 13:47:14.945351 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.945325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" event={"ID":"327a19d2-ed61-4610-b98c-a7c88b6049eb","Type":"ContainerStarted","Data":"e8de7795e3624845c57743fae27342e8cf5efe67b4cb8299d9416a14dadbd5e6"} Jul 24 13:47:14.945486 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.945377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:14.945486 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.945393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" event={"ID":"327a19d2-ed61-4610-b98c-a7c88b6049eb","Type":"ContainerStarted","Data":"e8e07245d746a0cdce53fd58bcbee8579a0d7c863d0cf0ce9961516eeb4e6e62"} Jul 24 13:47:14.946835 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.946810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" event={"ID":"4c52dabe-b5cc-4328-b714-9027e4bac2e0","Type":"ContainerStarted","Data":"fc1e27a7d478f7f15e30d84744e34ca1a14475fbbab7e25ecaaf0694227428d5"} Jul 24 13:47:14.946951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.946854 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" event={"ID":"4c52dabe-b5cc-4328-b714-9027e4bac2e0","Type":"ContainerStarted","Data":"45d05b25c44f6036bc9a9fee57c144160f6c6896ad4fbcb933eb4337f7738c1d"} Jul 24 13:47:14.946951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.946880 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:14.948108 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.948085 2581 generic.go:356] "Generic (PLEG): container finished" podID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" containerID="e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033" exitCode=0 Jul 24 13:47:14.948198 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.948120 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:14.948198 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.948173 2581 scope.go:117] "RemoveContainer" containerID="e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033" Jul 24 13:47:14.949535 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.949511 2581 generic.go:356] "Generic (PLEG): container finished" podID="0f76919f-62af-4134-8344-03b767239562" containerID="49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1" exitCode=0 Jul 24 13:47:14.949648 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.949542 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:14.951490 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.951442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" event={"ID":"3de03b83-b626-4236-9f5a-62434c87281d","Type":"ContainerStarted","Data":"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd"} Jul 24 13:47:14.951649 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.951505 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" event={"ID":"3de03b83-b626-4236-9f5a-62434c87281d","Type":"ContainerStarted","Data":"aaf1468e8ba7fec1d715bed5f8eae5edd8a31b02b8632278bd6bcf791e851d28"} Jul 24 13:47:14.951649 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.951632 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:14.956715 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.956697 2581 scope.go:117] "RemoveContainer" containerID="e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033" Jul 24 13:47:14.956994 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.956976 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033\": container with ID starting with e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033 not found: ID does not exist" containerID="e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033" Jul 24 13:47:14.957036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.957004 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033"} err="failed to get container status \"e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033\": rpc error: code = NotFound desc = could not find container \"e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033\": container with ID starting with e5ad89204f924d16f46d81032d1a9347b9c3beee0de4db4456393367afd35033 not found: ID does not exist" Jul 24 13:47:14.957036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.957024 2581 scope.go:117] "RemoveContainer" containerID="49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1" Jul 24 13:47:14.960440 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.960396 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5ntqw" podStartSLOduration=2.384011538 podStartE2EDuration="23.960381761s" totalImagesPullingTime="21.576370223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:14.959204277 +0000 UTC m=+528.555768351" watchObservedRunningTime="2026-07-24 13:47:14.960381761 +0000 UTC m=+528.556945816" Jul 24 13:47:14.961455 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.961428 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:14.966262 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.966238 2581 scope.go:117] "RemoveContainer" containerID="49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1" Jul 24 13:47:14.966700 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:14.966653 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1\": container with ID starting with 49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1 not found: ID does not exist" containerID="49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1" Jul 24 13:47:14.966814 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.966694 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1"} err="failed to get container status \"49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1\": rpc error: code = NotFound desc = could not find container \"49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1\": container with ID starting with 49fb65a07cc94d8f1a73b6a452dc13f84d9163dadb543690418a54bcbbd813e1 not found: ID does not exist" Jul 24 13:47:14.981943 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:14.981840 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" podStartSLOduration=12.981824421 podStartE2EDuration="12.981824421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:14.981626201 +0000 UTC m=+528.578190257" watchObservedRunningTime="2026-07-24 13:47:14.981824421 +0000 UTC m=+528.578388466" Jul 24 13:47:15.002110 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:15.002059 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" podStartSLOduration=13.002039579 podStartE2EDuration="13.002039579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:15.000637909 +0000 UTC m=+528.597201966" watchObservedRunningTime="2026-07-24 13:47:15.002039579 +0000 UTC m=+528.598603634" Jul 24 13:47:15.002524 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:15.002500 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0f76919f-62af-4134-8344-03b767239562" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:15.022237 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:15.022189 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" podStartSLOduration=13.022172548 podStartE2EDuration="13.022172548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:15.020873797 +0000 UTC m=+528.617437874" watchObservedRunningTime="2026-07-24 13:47:15.022172548 +0000 UTC m=+528.618736603" Jul 24 13:47:15.152003 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:15.151969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:15.252490 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:15.252401 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:16.876993 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.876955 2581 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-5th6w_kuadrant-system_6a43e0d8-4d26-4c5e-b31b-7c83391a528f_0(0e14590b1e376e5585a7bb39c6a426d626148c7b6e8834bd51a293ccbf60a503): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-5th6w to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 24 13:47:16.877298 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.877025 2581 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-5th6w_kuadrant-system_6a43e0d8-4d26-4c5e-b31b-7c83391a528f_0(0e14590b1e376e5585a7bb39c6a426d626148c7b6e8834bd51a293ccbf60a503): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-5th6w to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:16.914285 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.914241 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sr9sl\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0fe6ae90-2c1b-4b1d-8fbf-ecda51e14f2a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sr9sl" Jul 24 13:47:16.915983 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.915955 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wk2lw\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="0f76919f-62af-4134-8344-03b767239562" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wk2lw" Jul 24 13:47:16.961474 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:16.961444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:16.963821 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.963784 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="6a43e0d8-4d26-4c5e-b31b-7c83391a528f" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:16.970741 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:16.970708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:16.972900 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:16.972866 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="6a43e0d8-4d26-4c5e-b31b-7c83391a528f" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:17.009157 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.009122 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lk8h\" (UniqueName: \"kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h\") pod \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " Jul 24 13:47:17.009391 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.009261 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume\") pod \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\" (UID: \"6a43e0d8-4d26-4c5e-b31b-7c83391a528f\") " Jul 24 13:47:17.009535 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.009509 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6a43e0d8-4d26-4c5e-b31b-7c83391a528f" (UID: "6a43e0d8-4d26-4c5e-b31b-7c83391a528f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:47:17.011328 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.011292 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h" (OuterVolumeSpecName: "kube-api-access-9lk8h") pod "6a43e0d8-4d26-4c5e-b31b-7c83391a528f" (UID: "6a43e0d8-4d26-4c5e-b31b-7c83391a528f"). InnerVolumeSpecName "kube-api-access-9lk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:17.109846 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.109805 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-extensions-socket-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:17.109846 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.109837 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lk8h\" (UniqueName: \"kubernetes.io/projected/6a43e0d8-4d26-4c5e-b31b-7c83391a528f-kube-api-access-9lk8h\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:17.966681 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:17.966646 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:17.968839 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:17.968801 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="6a43e0d8-4d26-4c5e-b31b-7c83391a528f" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:17.974847 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:17.974816 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5th6w\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="6a43e0d8-4d26-4c5e-b31b-7c83391a528f" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5th6w" Jul 24 13:47:18.915374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:18.915341 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a43e0d8-4d26-4c5e-b31b-7c83391a528f" path="/var/lib/kubelet/pods/6a43e0d8-4d26-4c5e-b31b-7c83391a528f/volumes" Jul 24 13:47:25.959100 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:25.959059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:25.959642 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:25.959150 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:25.959642 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:25.959306 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jn88c" Jul 24 13:47:26.029776 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.029740 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:26.030008 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.029982 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" podUID="3de03b83-b626-4236-9f5a-62434c87281d" containerName="manager" containerID="cri-o://fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd" gracePeriod=10 Jul 24 13:47:26.250510 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.250430 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:47:26.253826 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.253804 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.266727 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.266697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:47:26.288688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.288660 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:26.380831 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.380799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume\") pod \"3de03b83-b626-4236-9f5a-62434c87281d\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " Jul 24 13:47:26.381082 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.380892 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjvqn\" (UniqueName: \"kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn\") pod \"3de03b83-b626-4236-9f5a-62434c87281d\" (UID: \"3de03b83-b626-4236-9f5a-62434c87281d\") " Jul 24 13:47:26.381082 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.380987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.381082 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.381024 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2lm\" (UniqueName: \"kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.381294 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.381262 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3de03b83-b626-4236-9f5a-62434c87281d" (UID: "3de03b83-b626-4236-9f5a-62434c87281d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:47:26.383072 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.383043 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn" (OuterVolumeSpecName: "kube-api-access-rjvqn") pod "3de03b83-b626-4236-9f5a-62434c87281d" (UID: "3de03b83-b626-4236-9f5a-62434c87281d"). InnerVolumeSpecName "kube-api-access-rjvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:26.481635 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.481557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2lm\" (UniqueName: \"kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.481821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.481682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.481821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.481724 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvqn\" (UniqueName: \"kubernetes.io/projected/3de03b83-b626-4236-9f5a-62434c87281d-kube-api-access-rjvqn\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:26.481821 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.481735 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de03b83-b626-4236-9f5a-62434c87281d-extensions-socket-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:26.482065 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.482045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.493241 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.493210 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2lm\" (UniqueName: \"kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm\") pod \"kuadrant-operator-controller-manager-86878c769f-wf48z\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.564397 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.564296 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:26.706952 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:26.706920 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:47:26.710468 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:26.710441 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c9608a_efdd_450c_9600_61a77218dddd.slice/crio-626fb46f631db5399d84060d789086a2a12917b2142e260dc783c198432f815f WatchSource:0}: Error finding container 626fb46f631db5399d84060d789086a2a12917b2142e260dc783c198432f815f: Status 404 returned error can't find the container with id 626fb46f631db5399d84060d789086a2a12917b2142e260dc783c198432f815f Jul 24 13:47:27.007617 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.007568 2581 generic.go:356] "Generic (PLEG): container finished" podID="3de03b83-b626-4236-9f5a-62434c87281d" containerID="fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd" exitCode=0 Jul 24 13:47:27.008071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.007652 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" Jul 24 13:47:27.008071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.007655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" event={"ID":"3de03b83-b626-4236-9f5a-62434c87281d","Type":"ContainerDied","Data":"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd"} Jul 24 13:47:27.008071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.007715 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd" event={"ID":"3de03b83-b626-4236-9f5a-62434c87281d","Type":"ContainerDied","Data":"aaf1468e8ba7fec1d715bed5f8eae5edd8a31b02b8632278bd6bcf791e851d28"} Jul 24 13:47:27.008071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.007744 2581 scope.go:117] "RemoveContainer" containerID="fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd" Jul 24 13:47:27.009370 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.009341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" event={"ID":"d7c9608a-efdd-450c-9600-61a77218dddd","Type":"ContainerStarted","Data":"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4"} Jul 24 13:47:27.009473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.009395 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" event={"ID":"d7c9608a-efdd-450c-9600-61a77218dddd","Type":"ContainerStarted","Data":"626fb46f631db5399d84060d789086a2a12917b2142e260dc783c198432f815f"} Jul 24 13:47:27.009473 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.009450 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:27.016164 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.016139 2581 scope.go:117] "RemoveContainer" containerID="fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd" Jul 24 13:47:27.016402 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:47:27.016384 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd\": container with ID starting with fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd not found: ID does not exist" containerID="fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd" Jul 24 13:47:27.016463 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.016413 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd"} err="failed to get container status \"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd\": rpc error: code = NotFound desc = could not find container \"fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd\": container with ID starting with fd3830ea166b84dfd8d1d92619e84bef616a4333353a54184e357fcf03cae0dd not found: ID does not exist" Jul 24 13:47:27.033213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.033165 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" podStartSLOduration=1.033148954 podStartE2EDuration="1.033148954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:47:27.032399607 +0000 UTC m=+540.628963662" watchObservedRunningTime="2026-07-24 13:47:27.033148954 +0000 UTC m=+540.629713009" Jul 24 13:47:27.045265 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.045233 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:27.054767 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:27.054741 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dz7kd"] Jul 24 13:47:28.915993 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:28.915961 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de03b83-b626-4236-9f5a-62434c87281d" path="/var/lib/kubelet/pods/3de03b83-b626-4236-9f5a-62434c87281d/volumes" Jul 24 13:47:38.017909 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:38.017879 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:47:38.071217 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:38.071185 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:38.071435 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:38.071394 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" podUID="327a19d2-ed61-4610-b98c-a7c88b6049eb" containerName="manager" containerID="cri-o://e8de7795e3624845c57743fae27342e8cf5efe67b4cb8299d9416a14dadbd5e6" gracePeriod=10 Jul 24 13:47:39.067359 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.067284 2581 generic.go:356] "Generic (PLEG): container finished" podID="327a19d2-ed61-4610-b98c-a7c88b6049eb" containerID="e8de7795e3624845c57743fae27342e8cf5efe67b4cb8299d9416a14dadbd5e6" exitCode=0 Jul 24 13:47:39.067746 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.067356 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" event={"ID":"327a19d2-ed61-4610-b98c-a7c88b6049eb","Type":"ContainerDied","Data":"e8de7795e3624845c57743fae27342e8cf5efe67b4cb8299d9416a14dadbd5e6"} Jul 24 13:47:39.186219 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.186181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:39.290389 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.290348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:39.445551 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.445519 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:39.589947 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.589913 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume\") pod \"327a19d2-ed61-4610-b98c-a7c88b6049eb\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " Jul 24 13:47:39.590125 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.590005 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbnx5\" (UniqueName: \"kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5\") pod \"327a19d2-ed61-4610-b98c-a7c88b6049eb\" (UID: \"327a19d2-ed61-4610-b98c-a7c88b6049eb\") " Jul 24 13:47:39.590386 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.590351 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "327a19d2-ed61-4610-b98c-a7c88b6049eb" (UID: "327a19d2-ed61-4610-b98c-a7c88b6049eb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:47:39.592062 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.592028 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5" (OuterVolumeSpecName: "kube-api-access-wbnx5") pod "327a19d2-ed61-4610-b98c-a7c88b6049eb" (UID: "327a19d2-ed61-4610-b98c-a7c88b6049eb"). InnerVolumeSpecName "kube-api-access-wbnx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:39.690886 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.690851 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/327a19d2-ed61-4610-b98c-a7c88b6049eb-extensions-socket-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:39.690886 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:39.690879 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnx5\" (UniqueName: \"kubernetes.io/projected/327a19d2-ed61-4610-b98c-a7c88b6049eb-kube-api-access-wbnx5\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:47:40.073450 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.073380 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" Jul 24 13:47:40.073450 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.073424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7" event={"ID":"327a19d2-ed61-4610-b98c-a7c88b6049eb","Type":"ContainerDied","Data":"e8e07245d746a0cdce53fd58bcbee8579a0d7c863d0cf0ce9961516eeb4e6e62"} Jul 24 13:47:40.073971 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.073485 2581 scope.go:117] "RemoveContainer" containerID="e8de7795e3624845c57743fae27342e8cf5efe67b4cb8299d9416a14dadbd5e6" Jul 24 13:47:40.094322 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.094290 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:40.100367 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.100339 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ps8n7"] Jul 24 13:47:40.915314 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:40.915285 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="327a19d2-ed61-4610-b98c-a7c88b6049eb" path="/var/lib/kubelet/pods/327a19d2-ed61-4610-b98c-a7c88b6049eb/volumes" Jul 24 13:47:41.507182 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:41.507153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:42.415325 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:42.415297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:42.600206 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:42.600174 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:52.309642 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:52.309567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:55.270832 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.270801 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:55.276674 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.276648 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:47:55.276959 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.276947 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="327a19d2-ed61-4610-b98c-a7c88b6049eb" containerName="manager" Jul 24 13:47:55.277004 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.276960 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="327a19d2-ed61-4610-b98c-a7c88b6049eb" containerName="manager" Jul 24 13:47:55.277004 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.276983 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de03b83-b626-4236-9f5a-62434c87281d" containerName="manager" Jul 24 13:47:55.277004 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.276988 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de03b83-b626-4236-9f5a-62434c87281d" containerName="manager" Jul 24 13:47:55.277100 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.277034 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de03b83-b626-4236-9f5a-62434c87281d" containerName="manager" Jul 24 13:47:55.277100 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.277050 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="327a19d2-ed61-4610-b98c-a7c88b6049eb" containerName="manager" Jul 24 13:47:55.278963 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.278949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.281213 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.281191 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 13:47:55.289342 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.289320 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:47:55.376965 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.376937 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:47:55.381326 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.381302 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:55.418871 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.418840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfw4\" (UniqueName: \"kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.419040 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.418900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.442512 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.442479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:55.519598 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.519549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfw4\" (UniqueName: \"kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.519781 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.519647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.520222 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.520205 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.527785 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.527723 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfw4\" (UniqueName: \"kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4\") pod \"limitador-limitador-64c94fd64f-kfdpw\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.589419 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.589390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:47:55.719576 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:55.719401 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:47:55.726062 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:55.726024 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228b157d_9b88_4d4b_9097_7324a224c563.slice/crio-c176d70326e2c7faf53e007d33b27c78d551f64beca14568e674f88afd8f8680 WatchSource:0}: Error finding container c176d70326e2c7faf53e007d33b27c78d551f64beca14568e674f88afd8f8680: Status 404 returned error can't find the container with id c176d70326e2c7faf53e007d33b27c78d551f64beca14568e674f88afd8f8680 Jul 24 13:47:56.068651 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.068617 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:47:56.071854 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.071831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:47:56.074012 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.073986 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjlpp\"" Jul 24 13:47:56.077780 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.077752 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:47:56.103364 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.103328 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:56.142325 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.142291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" event={"ID":"228b157d-9b88-4d4b-9097-7324a224c563","Type":"ContainerStarted","Data":"c176d70326e2c7faf53e007d33b27c78d551f64beca14568e674f88afd8f8680"} Jul 24 13:47:56.154734 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.154703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:47:56.227846 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.227812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8cv\" (UniqueName: \"kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv\") pod \"authorino-58cd955675-cxx54\" (UID: \"6c67912c-980a-45b6-a562-345901ec5280\") " pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:47:56.329076 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.328988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8cv\" (UniqueName: \"kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv\") pod \"authorino-58cd955675-cxx54\" (UID: \"6c67912c-980a-45b6-a562-345901ec5280\") " pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:47:56.337997 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.337968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8cv\" (UniqueName: \"kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv\") pod \"authorino-58cd955675-cxx54\" (UID: \"6c67912c-980a-45b6-a562-345901ec5280\") " pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:47:56.382726 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.382689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:47:56.563205 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:56.561362 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:47:56.566646 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:47:56.566610 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c67912c_980a_45b6_a562_345901ec5280.slice/crio-c04d76f45cdb54b606886ecb2b3c316ea2b4bf79c9810ea33b38211a22949c74 WatchSource:0}: Error finding container c04d76f45cdb54b606886ecb2b3c316ea2b4bf79c9810ea33b38211a22949c74: Status 404 returned error can't find the container with id c04d76f45cdb54b606886ecb2b3c316ea2b4bf79c9810ea33b38211a22949c74 Jul 24 13:47:57.149833 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:47:57.149792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxx54" event={"ID":"6c67912c-980a-45b6-a562-345901ec5280","Type":"ContainerStarted","Data":"c04d76f45cdb54b606886ecb2b3c316ea2b4bf79c9810ea33b38211a22949c74"} Jul 24 13:48:00.685200 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:00.685168 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:48:01.174872 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:01.174819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" event={"ID":"228b157d-9b88-4d4b-9097-7324a224c563","Type":"ContainerStarted","Data":"caf4bbd3e03e79eb295f6076a52295ba02fe3cdec3f9146113658878415a3458"} Jul 24 13:48:01.175165 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:01.175129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:48:01.176472 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:01.176450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxx54" event={"ID":"6c67912c-980a-45b6-a562-345901ec5280","Type":"ContainerStarted","Data":"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758"} Jul 24 13:48:01.192171 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:01.192116 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" podStartSLOduration=1.5632643750000002 podStartE2EDuration="6.192099889s" totalImagesPullingTime="4.628835514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:01.190371731 +0000 UTC m=+574.786935786" watchObservedRunningTime="2026-07-24 13:48:01.192099889 +0000 UTC m=+574.788663944" Jul 24 13:48:01.204934 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:01.204885 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cxx54" podStartSLOduration=1.404830572 podStartE2EDuration="5.20487024s" totalImagesPullingTime="3.800039668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:01.203204226 +0000 UTC m=+574.799768386" watchObservedRunningTime="2026-07-24 13:48:01.20487024 +0000 UTC m=+574.801434294" Jul 24 13:48:02.180468 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:02.180429 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cxx54" podUID="6c67912c-980a-45b6-a562-345901ec5280" containerName="authorino" containerID="cri-o://3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758" gracePeriod=30 Jul 24 13:48:02.433153 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:02.433079 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:48:02.583826 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:02.583785 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x8cv\" (UniqueName: \"kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv\") pod \"6c67912c-980a-45b6-a562-345901ec5280\" (UID: \"6c67912c-980a-45b6-a562-345901ec5280\") " Jul 24 13:48:02.585889 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:02.585866 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv" (OuterVolumeSpecName: "kube-api-access-9x8cv") pod "6c67912c-980a-45b6-a562-345901ec5280" (UID: "6c67912c-980a-45b6-a562-345901ec5280"). InnerVolumeSpecName "kube-api-access-9x8cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:48:02.684619 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:02.684506 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8cv\" (UniqueName: \"kubernetes.io/projected/6c67912c-980a-45b6-a562-345901ec5280-kube-api-access-9x8cv\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:03.185461 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.185429 2581 generic.go:356] "Generic (PLEG): container finished" podID="6c67912c-980a-45b6-a562-345901ec5280" containerID="3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758" exitCode=0 Jul 24 13:48:03.185923 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.185468 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxx54" Jul 24 13:48:03.185923 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.185476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxx54" event={"ID":"6c67912c-980a-45b6-a562-345901ec5280","Type":"ContainerDied","Data":"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758"} Jul 24 13:48:03.185923 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.185507 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxx54" event={"ID":"6c67912c-980a-45b6-a562-345901ec5280","Type":"ContainerDied","Data":"c04d76f45cdb54b606886ecb2b3c316ea2b4bf79c9810ea33b38211a22949c74"} Jul 24 13:48:03.185923 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.185523 2581 scope.go:117] "RemoveContainer" containerID="3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758" Jul 24 13:48:03.193875 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.193852 2581 scope.go:117] "RemoveContainer" containerID="3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758" Jul 24 13:48:03.194179 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:48:03.194158 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758\": container with ID starting with 3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758 not found: ID does not exist" containerID="3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758" Jul 24 13:48:03.194246 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.194186 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758"} err="failed to get container status \"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758\": rpc error: code = NotFound desc = could not find container \"3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758\": container with ID starting with 3d25491bbbb6233183622d2530bf7bdb5329d56e2978340b85f114c7d5056758 not found: ID does not exist" Jul 24 13:48:03.202375 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.202345 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:48:03.205551 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:03.205531 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxx54"] Jul 24 13:48:04.916895 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:04.916862 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c67912c-980a-45b6-a562-345901ec5280" path="/var/lib/kubelet/pods/6c67912c-980a-45b6-a562-345901ec5280/volumes" Jul 24 13:48:11.696548 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:11.696516 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:48:11.697115 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:11.696761 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" podUID="228b157d-9b88-4d4b-9097-7324a224c563" containerName="limitador" containerID="cri-o://caf4bbd3e03e79eb295f6076a52295ba02fe3cdec3f9146113658878415a3458" gracePeriod=30 Jul 24 13:48:11.697564 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:11.697534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:48:12.227880 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.227837 2581 generic.go:356] "Generic (PLEG): container finished" podID="228b157d-9b88-4d4b-9097-7324a224c563" containerID="caf4bbd3e03e79eb295f6076a52295ba02fe3cdec3f9146113658878415a3458" exitCode=0 Jul 24 13:48:12.228057 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.227908 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" event={"ID":"228b157d-9b88-4d4b-9097-7324a224c563","Type":"ContainerDied","Data":"caf4bbd3e03e79eb295f6076a52295ba02fe3cdec3f9146113658878415a3458"} Jul 24 13:48:12.258917 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.258843 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:48:12.356971 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.356931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file\") pod \"228b157d-9b88-4d4b-9097-7324a224c563\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " Jul 24 13:48:12.356971 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.356978 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvfw4\" (UniqueName: \"kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4\") pod \"228b157d-9b88-4d4b-9097-7324a224c563\" (UID: \"228b157d-9b88-4d4b-9097-7324a224c563\") " Jul 24 13:48:12.357312 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.357285 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file" (OuterVolumeSpecName: "config-file") pod "228b157d-9b88-4d4b-9097-7324a224c563" (UID: "228b157d-9b88-4d4b-9097-7324a224c563"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:48:12.359050 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.359025 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4" (OuterVolumeSpecName: "kube-api-access-vvfw4") pod "228b157d-9b88-4d4b-9097-7324a224c563" (UID: "228b157d-9b88-4d4b-9097-7324a224c563"). InnerVolumeSpecName "kube-api-access-vvfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:48:12.458086 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.458040 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/228b157d-9b88-4d4b-9097-7324a224c563-config-file\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:12.458086 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:12.458080 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfw4\" (UniqueName: \"kubernetes.io/projected/228b157d-9b88-4d4b-9097-7324a224c563-kube-api-access-vvfw4\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:13.233115 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:13.233040 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" Jul 24 13:48:13.233551 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:13.233033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kfdpw" event={"ID":"228b157d-9b88-4d4b-9097-7324a224c563","Type":"ContainerDied","Data":"c176d70326e2c7faf53e007d33b27c78d551f64beca14568e674f88afd8f8680"} Jul 24 13:48:13.233551 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:13.233183 2581 scope.go:117] "RemoveContainer" containerID="caf4bbd3e03e79eb295f6076a52295ba02fe3cdec3f9146113658878415a3458" Jul 24 13:48:13.248788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:13.248755 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:48:13.252505 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:13.252477 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kfdpw"] Jul 24 13:48:14.916206 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:14.916169 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228b157d-9b88-4d4b-9097-7324a224c563" path="/var/lib/kubelet/pods/228b157d-9b88-4d4b-9097-7324a224c563/volumes" Jul 24 13:48:16.134021 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.133985 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:16.233948 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.233915 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:16.615280 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615231 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f"] Jul 24 13:48:16.615762 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615742 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228b157d-9b88-4d4b-9097-7324a224c563" containerName="limitador" Jul 24 13:48:16.615762 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615762 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="228b157d-9b88-4d4b-9097-7324a224c563" containerName="limitador" Jul 24 13:48:16.615939 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615773 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c67912c-980a-45b6-a562-345901ec5280" containerName="authorino" Jul 24 13:48:16.615939 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615782 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c67912c-980a-45b6-a562-345901ec5280" containerName="authorino" Jul 24 13:48:16.615939 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615855 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="228b157d-9b88-4d4b-9097-7324a224c563" containerName="limitador" Jul 24 13:48:16.615939 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.615868 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c67912c-980a-45b6-a562-345901ec5280" containerName="authorino" Jul 24 13:48:16.620978 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.620951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.624014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.623979 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7zf6x\"" Jul 24 13:48:16.624014 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.623992 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 13:48:16.624325 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.624305 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 13:48:16.624424 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.624339 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 13:48:16.625190 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.625169 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f"] Jul 24 13:48:16.644728 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.644700 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:16.686918 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.686890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4zf\" (UniqueName: \"kubernetes.io/projected/32aefa77-9065-449a-a169-61438c1284e8-kube-api-access-8j4zf\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.687051 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.686934 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32aefa77-9065-449a-a169-61438c1284e8-data\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.787845 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.787799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32aefa77-9065-449a-a169-61438c1284e8-data\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.788046 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.787907 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4zf\" (UniqueName: \"kubernetes.io/projected/32aefa77-9065-449a-a169-61438c1284e8-kube-api-access-8j4zf\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.788181 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.788161 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/32aefa77-9065-449a-a169-61438c1284e8-data\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.796806 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.796764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4zf\" (UniqueName: \"kubernetes.io/projected/32aefa77-9065-449a-a169-61438c1284e8-kube-api-access-8j4zf\") pod \"postgres-69fb4594cf-hfn5f\" (UID: \"32aefa77-9065-449a-a169-61438c1284e8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:16.934618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:16.934567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:17.080058 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:17.080016 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f"] Jul 24 13:48:17.082630 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:48:17.082601 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aefa77_9065_449a_a169_61438c1284e8.slice/crio-76181536b003a42de5ce69a41fede89b7dae16b837ef269430671f545c155d9a WatchSource:0}: Error finding container 76181536b003a42de5ce69a41fede89b7dae16b837ef269430671f545c155d9a: Status 404 returned error can't find the container with id 76181536b003a42de5ce69a41fede89b7dae16b837ef269430671f545c155d9a Jul 24 13:48:17.255036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:17.254945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" event={"ID":"32aefa77-9065-449a-a169-61438c1284e8","Type":"ContainerStarted","Data":"76181536b003a42de5ce69a41fede89b7dae16b837ef269430671f545c155d9a"} Jul 24 13:48:23.291145 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:23.291103 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" event={"ID":"32aefa77-9065-449a-a169-61438c1284e8","Type":"ContainerStarted","Data":"e3ac6d5a8aa266675f1fc39e4307fb62671d4eadd1c31be6aec677a5e4165e01"} Jul 24 13:48:23.291631 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:23.291218 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:23.307711 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:23.307661 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" podStartSLOduration=2.130345922 podStartE2EDuration="7.307644299s" totalImagesPullingTime="5.177298377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:23.306369089 +0000 UTC m=+596.902933146" watchObservedRunningTime="2026-07-24 13:48:23.307644299 +0000 UTC m=+596.904208353" Jul 24 13:48:29.324441 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.324407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfn5f" Jul 24 13:48:29.640830 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.640741 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:29.873936 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.873900 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vcrpm"] Jul 24 13:48:29.883577 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.883540 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:29.884274 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.884254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vcrpm"] Jul 24 13:48:29.884367 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.884355 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:29.886494 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.886473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjlpp\"" Jul 24 13:48:29.981292 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:29.981243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smgw\" (UniqueName: \"kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw\") pod \"authorino-5d4b67c446-vcrpm\" (UID: \"45ea257b-31f3-4f51-8694-1c7e2c22aa9b\") " pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:30.051223 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.051194 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vcrpm"] Jul 24 13:48:30.051450 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:48:30.051431 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6smgw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-vcrpm" podUID="45ea257b-31f3-4f51-8694-1c7e2c22aa9b" Jul 24 13:48:30.082044 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.082000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smgw\" (UniqueName: \"kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw\") pod \"authorino-5d4b67c446-vcrpm\" (UID: \"45ea257b-31f3-4f51-8694-1c7e2c22aa9b\") " pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:30.083528 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.083505 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-bbvxx"] Jul 24 13:48:30.087062 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.087044 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.089683 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.089580 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 13:48:30.101348 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.101311 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-bbvxx"] Jul 24 13:48:30.103228 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.103202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smgw\" (UniqueName: \"kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw\") pod \"authorino-5d4b67c446-vcrpm\" (UID: \"45ea257b-31f3-4f51-8694-1c7e2c22aa9b\") " pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:30.140417 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.140381 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-bbvxx"] Jul 24 13:48:30.140643 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:48:30.140624 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qzgpx tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-qzgpx tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-bbvxx" podUID="b4e91f8b-1f6b-4f33-bd17-d9068002d515" Jul 24 13:48:30.170800 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.170768 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:48:30.174568 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.174551 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.179369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.179341 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:48:30.182538 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.182402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.182538 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.182459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cnkp\" (UniqueName: \"kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.182538 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.182490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgpx\" (UniqueName: \"kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.182747 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.182566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.283609 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.283548 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.283788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.283627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnkp\" (UniqueName: \"kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.283788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.283657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgpx\" (UniqueName: \"kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.283788 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.283685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.286279 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.286251 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.286416 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.286251 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.291115 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.291086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cnkp\" (UniqueName: \"kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp\") pod \"authorino-5cfc6c5fb5-pqcpn\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.291234 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.291089 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgpx\" (UniqueName: \"kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx\") pod \"authorino-76877994-bbvxx\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.334681 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.334642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:30.335156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.334642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.343480 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.343444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:30.346178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.346158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:30.384402 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.384368 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6smgw\" (UniqueName: \"kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw\") pod \"45ea257b-31f3-4f51-8694-1c7e2c22aa9b\" (UID: \"45ea257b-31f3-4f51-8694-1c7e2c22aa9b\") " Jul 24 13:48:30.384618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.384412 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzgpx\" (UniqueName: \"kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx\") pod \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " Jul 24 13:48:30.384618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.384444 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert\") pod \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\" (UID: \"b4e91f8b-1f6b-4f33-bd17-d9068002d515\") " Jul 24 13:48:30.386687 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.386653 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx" (OuterVolumeSpecName: "kube-api-access-qzgpx") pod "b4e91f8b-1f6b-4f33-bd17-d9068002d515" (UID: "b4e91f8b-1f6b-4f33-bd17-d9068002d515"). InnerVolumeSpecName "kube-api-access-qzgpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:48:30.386835 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.386661 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b4e91f8b-1f6b-4f33-bd17-d9068002d515" (UID: "b4e91f8b-1f6b-4f33-bd17-d9068002d515"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:48:30.386835 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.386700 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw" (OuterVolumeSpecName: "kube-api-access-6smgw") pod "45ea257b-31f3-4f51-8694-1c7e2c22aa9b" (UID: "45ea257b-31f3-4f51-8694-1c7e2c22aa9b"). InnerVolumeSpecName "kube-api-access-6smgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:48:30.485149 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.485055 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:48:30.485299 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.485187 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smgw\" (UniqueName: \"kubernetes.io/projected/45ea257b-31f3-4f51-8694-1c7e2c22aa9b-kube-api-access-6smgw\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:30.485299 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.485209 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgpx\" (UniqueName: \"kubernetes.io/projected/b4e91f8b-1f6b-4f33-bd17-d9068002d515-kube-api-access-qzgpx\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:30.485299 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.485219 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4e91f8b-1f6b-4f33-bd17-d9068002d515-tls-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:30.619655 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:30.619626 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:48:30.622052 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:48:30.622025 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod396cab13_7532_45d9_a187_78fe0f5c5d1b.slice/crio-554611144eb53a2c927b11dbd560b6ef0f13e2345e4186d3bca2c02c3546d599 WatchSource:0}: Error finding container 554611144eb53a2c927b11dbd560b6ef0f13e2345e4186d3bca2c02c3546d599: Status 404 returned error can't find the container with id 554611144eb53a2c927b11dbd560b6ef0f13e2345e4186d3bca2c02c3546d599 Jul 24 13:48:31.340071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.339968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" event={"ID":"396cab13-7532-45d9-a187-78fe0f5c5d1b","Type":"ContainerStarted","Data":"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237"} Jul 24 13:48:31.340071 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.340011 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" event={"ID":"396cab13-7532-45d9-a187-78fe0f5c5d1b","Type":"ContainerStarted","Data":"554611144eb53a2c927b11dbd560b6ef0f13e2345e4186d3bca2c02c3546d599"} Jul 24 13:48:31.340605 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.340115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-bbvxx" Jul 24 13:48:31.340605 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.340136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-vcrpm" Jul 24 13:48:31.355210 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.355159 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" podStartSLOduration=0.991511716 podStartE2EDuration="1.355144203s" totalImagesPullingTime="363.632487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:31.354330397 +0000 UTC m=+604.950894451" watchObservedRunningTime="2026-07-24 13:48:31.355144203 +0000 UTC m=+604.951708258" Jul 24 13:48:31.376809 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.376777 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-bbvxx"] Jul 24 13:48:31.384960 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.384932 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-bbvxx"] Jul 24 13:48:31.408428 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.408386 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vcrpm"] Jul 24 13:48:31.414887 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:31.414851 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-vcrpm"] Jul 24 13:48:32.917120 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:32.917090 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ea257b-31f3-4f51-8694-1c7e2c22aa9b" path="/var/lib/kubelet/pods/45ea257b-31f3-4f51-8694-1c7e2c22aa9b/volumes" Jul 24 13:48:32.919689 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:32.917331 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e91f8b-1f6b-4f33-bd17-d9068002d515" path="/var/lib/kubelet/pods/b4e91f8b-1f6b-4f33-bd17-d9068002d515/volumes" Jul 24 13:48:36.133036 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.133002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:36.136654 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.136623 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:36.140214 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.140194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.142338 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.142312 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 13:48:36.142478 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.142430 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 13:48:36.142565 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.142547 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8s96w\"" Jul 24 13:48:36.150781 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.150752 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:36.231757 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.231718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntlk7\" (UniqueName: \"kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.231938 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.231777 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.324378 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.324342 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:48:36.326572 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.326550 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.332433 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.332411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntlk7\" (UniqueName: \"kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.332540 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.332456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.335234 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.335211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.336436 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.336415 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:48:36.343374 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.343347 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntlk7\" (UniqueName: \"kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7\") pod \"maas-controller-d889ff4-2m4rq\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.433195 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.433159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbd8\" (UniqueName: \"kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.433372 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.433251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.450746 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.450712 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:36.533982 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.533947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbd8\" (UniqueName: \"kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.534102 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.534002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.536622 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.536570 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.542699 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.542674 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbd8\" (UniqueName: \"kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8\") pod \"maas-controller-84b69666bc-4r4tr\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.585488 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.585366 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:36.588181 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:48:36.588152 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dd7e62_3cd0_4541_8277_2fcac94f5026.slice/crio-bf7c932578c26c12205f2911c409d1c475339e15beda85a4d267c2afef4927cd WatchSource:0}: Error finding container bf7c932578c26c12205f2911c409d1c475339e15beda85a4d267c2afef4927cd: Status 404 returned error can't find the container with id bf7c932578c26c12205f2911c409d1c475339e15beda85a4d267c2afef4927cd Jul 24 13:48:36.637112 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.637071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:36.767572 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:36.767544 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:48:36.769949 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:48:36.769921 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb880b3b6_ab06_4d50_aa91_70e4c3e53770.slice/crio-1d247aa36c1f979a3753375bba27d8a157be5041497e804cd5d5ca3edf740ac4 WatchSource:0}: Error finding container 1d247aa36c1f979a3753375bba27d8a157be5041497e804cd5d5ca3edf740ac4: Status 404 returned error can't find the container with id 1d247aa36c1f979a3753375bba27d8a157be5041497e804cd5d5ca3edf740ac4 Jul 24 13:48:37.372928 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:37.372893 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d889ff4-2m4rq" event={"ID":"67dd7e62-3cd0-4541-8277-2fcac94f5026","Type":"ContainerStarted","Data":"bf7c932578c26c12205f2911c409d1c475339e15beda85a4d267c2afef4927cd"} Jul 24 13:48:37.376702 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:37.376661 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-4r4tr" event={"ID":"b880b3b6-ab06-4d50-aa91-70e4c3e53770","Type":"ContainerStarted","Data":"1d247aa36c1f979a3753375bba27d8a157be5041497e804cd5d5ca3edf740ac4"} Jul 24 13:48:37.691717 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:37.691680 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:39.541306 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:39.541238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:39.643378 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:39.643284 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:48:40.393022 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.392986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-4r4tr" event={"ID":"b880b3b6-ab06-4d50-aa91-70e4c3e53770","Type":"ContainerStarted","Data":"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58"} Jul 24 13:48:40.393239 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.393051 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:40.394456 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.394424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d889ff4-2m4rq" event={"ID":"67dd7e62-3cd0-4541-8277-2fcac94f5026","Type":"ContainerStarted","Data":"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0"} Jul 24 13:48:40.394639 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.394487 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:40.411678 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.411625 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b69666bc-4r4tr" podStartSLOduration=1.827756791 podStartE2EDuration="4.411607688s" totalImagesPullingTime="2.583850897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:40.410553192 +0000 UTC m=+614.007117285" watchObservedRunningTime="2026-07-24 13:48:40.411607688 +0000 UTC m=+614.008171734" Jul 24 13:48:40.428370 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:40.428316 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d889ff4-2m4rq" podStartSLOduration=1.6557993770000001 podStartE2EDuration="4.4283004s" totalImagesPullingTime="2.772501023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:48:40.426938996 +0000 UTC m=+614.023503085" watchObservedRunningTime="2026-07-24 13:48:40.4283004 +0000 UTC m=+614.024864468" Jul 24 13:48:51.401916 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:51.401837 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:48:51.402376 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:51.401987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:51.470090 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:51.470059 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:51.470267 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:51.470244 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d889ff4-2m4rq" podUID="67dd7e62-3cd0-4541-8277-2fcac94f5026" containerName="manager" containerID="cri-o://0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0" gracePeriod=30 Jul 24 13:48:52.262603 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.262555 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:52.367437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.367352 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert\") pod \"67dd7e62-3cd0-4541-8277-2fcac94f5026\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " Jul 24 13:48:52.367437 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.367430 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntlk7\" (UniqueName: \"kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7\") pod \"67dd7e62-3cd0-4541-8277-2fcac94f5026\" (UID: \"67dd7e62-3cd0-4541-8277-2fcac94f5026\") " Jul 24 13:48:52.369630 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.369579 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "67dd7e62-3cd0-4541-8277-2fcac94f5026" (UID: "67dd7e62-3cd0-4541-8277-2fcac94f5026"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:48:52.369744 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.369681 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7" (OuterVolumeSpecName: "kube-api-access-ntlk7") pod "67dd7e62-3cd0-4541-8277-2fcac94f5026" (UID: "67dd7e62-3cd0-4541-8277-2fcac94f5026"). InnerVolumeSpecName "kube-api-access-ntlk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:48:52.452139 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.452102 2581 generic.go:356] "Generic (PLEG): container finished" podID="67dd7e62-3cd0-4541-8277-2fcac94f5026" containerID="0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0" exitCode=0 Jul 24 13:48:52.452540 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.452148 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d889ff4-2m4rq" Jul 24 13:48:52.452540 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.452190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d889ff4-2m4rq" event={"ID":"67dd7e62-3cd0-4541-8277-2fcac94f5026","Type":"ContainerDied","Data":"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0"} Jul 24 13:48:52.452540 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.452240 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d889ff4-2m4rq" event={"ID":"67dd7e62-3cd0-4541-8277-2fcac94f5026","Type":"ContainerDied","Data":"bf7c932578c26c12205f2911c409d1c475339e15beda85a4d267c2afef4927cd"} Jul 24 13:48:52.452540 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.452267 2581 scope.go:117] "RemoveContainer" containerID="0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0" Jul 24 13:48:52.460668 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.460646 2581 scope.go:117] "RemoveContainer" containerID="0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0" Jul 24 13:48:52.460967 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:48:52.460946 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0\": container with ID starting with 0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0 not found: ID does not exist" containerID="0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0" Jul 24 13:48:52.461023 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.460974 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0"} err="failed to get container status \"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0\": rpc error: code = NotFound desc = could not find container \"0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0\": container with ID starting with 0bc0a699d614fdb6f7c91f8662c0749ccc07106487dec7abe1774aafbdee9ba0 not found: ID does not exist" Jul 24 13:48:52.468186 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.468162 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67dd7e62-3cd0-4541-8277-2fcac94f5026-webhook-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:52.468186 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.468186 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntlk7\" (UniqueName: \"kubernetes.io/projected/67dd7e62-3cd0-4541-8277-2fcac94f5026-kube-api-access-ntlk7\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:48:52.474101 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.474075 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:52.477511 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.477484 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d889ff4-2m4rq"] Jul 24 13:48:52.915726 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:48:52.915695 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67dd7e62-3cd0-4541-8277-2fcac94f5026" path="/var/lib/kubelet/pods/67dd7e62-3cd0-4541-8277-2fcac94f5026/volumes" Jul 24 13:49:11.247694 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:11.247658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:11.340815 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:11.340782 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:11.919991 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:11.919958 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:12.276996 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.276905 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:12.404122 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.404081 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:12.508101 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.508060 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn"] Jul 24 13:49:12.508348 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.508332 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:12.508534 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.508517 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67dd7e62-3cd0-4541-8277-2fcac94f5026" containerName="manager" Jul 24 13:49:12.508637 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.508537 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67dd7e62-3cd0-4541-8277-2fcac94f5026" containerName="manager" Jul 24 13:49:12.508699 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.508646 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67dd7e62-3cd0-4541-8277-2fcac94f5026" containerName="manager" Jul 24 13:49:12.517603 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.517561 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.519907 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.519887 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 13:49:12.520046 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.519886 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jcjzt\"" Jul 24 13:49:12.526109 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.526076 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn"] Jul 24 13:49:12.527938 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.527872 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-spxjx"] Jul 24 13:49:12.531315 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.531292 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.544094 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.544063 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-spxjx"] Jul 24 13:49:12.640745 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.640709 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8cv\" (UniqueName: \"kubernetes.io/projected/58b9b0d5-27e3-4158-aa63-c9f89789c786-kube-api-access-6q8cv\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.640745 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.640749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b9b0d5-27e3-4158-aa63-c9f89789c786-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.641002 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.640799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdxt\" (UniqueName: \"kubernetes.io/projected/ebd03052-7832-4502-afd8-41270a2f3a39-kube-api-access-hkdxt\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.641002 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.640903 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd03052-7832-4502-afd8-41270a2f3a39-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.741364 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.741325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd03052-7832-4502-afd8-41270a2f3a39-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.741559 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.741375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8cv\" (UniqueName: \"kubernetes.io/projected/58b9b0d5-27e3-4158-aa63-c9f89789c786-kube-api-access-6q8cv\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.741559 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.741398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b9b0d5-27e3-4158-aa63-c9f89789c786-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.741559 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.741447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdxt\" (UniqueName: \"kubernetes.io/projected/ebd03052-7832-4502-afd8-41270a2f3a39-kube-api-access-hkdxt\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.742158 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.742134 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58b9b0d5-27e3-4158-aa63-c9f89789c786-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.742158 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.742150 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd03052-7832-4502-afd8-41270a2f3a39-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.756483 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.756455 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdxt\" (UniqueName: \"kubernetes.io/projected/ebd03052-7832-4502-afd8-41270a2f3a39-kube-api-access-hkdxt\") pod \"payload-pre-processing-687dbb6cf6-hhkjn\" (UID: \"ebd03052-7832-4502-afd8-41270a2f3a39\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.757938 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.757914 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8cv\" (UniqueName: \"kubernetes.io/projected/58b9b0d5-27e3-4158-aa63-c9f89789c786-kube-api-access-6q8cv\") pod \"payload-processing-5cc6cd444b-spxjx\" (UID: \"58b9b0d5-27e3-4158-aa63-c9f89789c786\") " pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.831032 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.830941 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:12.844449 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.844419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:12.984845 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.984802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn"] Jul 24 13:49:12.987899 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:49:12.987869 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd03052_7832_4502_afd8_41270a2f3a39.slice/crio-7c8af621aa68bc45e24aa4f163dc916d6c108e7ac1f28b27a44f58bf2a0ca2c8 WatchSource:0}: Error finding container 7c8af621aa68bc45e24aa4f163dc916d6c108e7ac1f28b27a44f58bf2a0ca2c8: Status 404 returned error can't find the container with id 7c8af621aa68bc45e24aa4f163dc916d6c108e7ac1f28b27a44f58bf2a0ca2c8 Jul 24 13:49:12.989912 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:12.989894 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:49:13.014646 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:13.014463 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-spxjx"] Jul 24 13:49:13.016759 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:49:13.016735 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b9b0d5_27e3_4158_aa63_c9f89789c786.slice/crio-1fcff717b384f06ceb079bfb3808f876a6e22ae401cb7cf4ad765e835644ff9b WatchSource:0}: Error finding container 1fcff717b384f06ceb079bfb3808f876a6e22ae401cb7cf4ad765e835644ff9b: Status 404 returned error can't find the container with id 1fcff717b384f06ceb079bfb3808f876a6e22ae401cb7cf4ad765e835644ff9b Jul 24 13:49:13.542092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:13.542053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" event={"ID":"58b9b0d5-27e3-4158-aa63-c9f89789c786","Type":"ContainerStarted","Data":"1fcff717b384f06ceb079bfb3808f876a6e22ae401cb7cf4ad765e835644ff9b"} Jul 24 13:49:13.543509 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:13.543473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" event={"ID":"ebd03052-7832-4502-afd8-41270a2f3a39","Type":"ContainerStarted","Data":"7c8af621aa68bc45e24aa4f163dc916d6c108e7ac1f28b27a44f58bf2a0ca2c8"} Jul 24 13:49:13.995885 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:13.995850 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:15.555674 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.555625 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" event={"ID":"58b9b0d5-27e3-4158-aa63-c9f89789c786","Type":"ContainerStarted","Data":"631117cdf9b5a84990113fe5d43fffc101cfbb839f06fa3f6851d1dc1d8ab31d"} Jul 24 13:49:15.556120 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.555700 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:15.557178 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.557152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" event={"ID":"ebd03052-7832-4502-afd8-41270a2f3a39","Type":"ContainerStarted","Data":"cdc63c18afde77c8a92621659a7d9e0c1a762bdb3e86584ccda5dd34332bb1ac"} Jul 24 13:49:15.557310 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.557291 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:15.572896 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.572843 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" podStartSLOduration=1.8375148110000001 podStartE2EDuration="3.572824846s" totalImagesPullingTime="1.735310035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:49:15.572224239 +0000 UTC m=+649.168788293" watchObservedRunningTime="2026-07-24 13:49:15.572824846 +0000 UTC m=+649.169388900" Jul 24 13:49:15.587255 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:15.587193 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" podStartSLOduration=1.816464455 podStartE2EDuration="3.587174894s" totalImagesPullingTime="1.770710439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:49:15.586012502 +0000 UTC m=+649.182576560" watchObservedRunningTime="2026-07-24 13:49:15.587174894 +0000 UTC m=+649.183738948" Jul 24 13:49:23.352156 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:23.352122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:23.441731 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:23.441699 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:26.563312 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:26.563281 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hhkjn" Jul 24 13:49:26.563735 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:26.563362 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-spxjx" Jul 24 13:49:32.729836 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.729794 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv"] Jul 24 13:49:32.734664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.734640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.737067 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.737044 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 13:49:32.737868 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.737847 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 13:49:32.737974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.737870 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 13:49:32.737974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.737946 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9qk2t\"" Jul 24 13:49:32.743723 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.743699 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv"] Jul 24 13:49:32.748363 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.748341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:32.811321 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.811321 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df8d823-0b4a-4fb0-af0a-756b25622b71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.811553 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.811553 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.811553 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.811553 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.811468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rxw\" (UniqueName: \"kubernetes.io/projected/9df8d823-0b4a-4fb0-af0a-756b25622b71-kube-api-access-26rxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912166 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rxw\" (UniqueName: \"kubernetes.io/projected/9df8d823-0b4a-4fb0-af0a-756b25622b71-kube-api-access-26rxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912242 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df8d823-0b4a-4fb0-af0a-756b25622b71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912355 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912787 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912787 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.912951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.912826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.914664 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.914639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df8d823-0b4a-4fb0-af0a-756b25622b71-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.915193 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.915174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df8d823-0b4a-4fb0-af0a-756b25622b71-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:32.919520 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:32.919489 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rxw\" (UniqueName: \"kubernetes.io/projected/9df8d823-0b4a-4fb0-af0a-756b25622b71-kube-api-access-26rxw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv\" (UID: \"9df8d823-0b4a-4fb0-af0a-756b25622b71\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:33.045728 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:33.045638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:33.196045 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:33.196019 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv"] Jul 24 13:49:33.645324 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:33.645280 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" event={"ID":"9df8d823-0b4a-4fb0-af0a-756b25622b71","Type":"ContainerStarted","Data":"38a1489e46fff246573da9e078a255dd77c87b52738732217096135f4e411d0f"} Jul 24 13:49:34.651641 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:34.651539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" event={"ID":"9df8d823-0b4a-4fb0-af0a-756b25622b71","Type":"ContainerStarted","Data":"2b5bebb6e09bf465d91271db61b49050ef1b5a21b81f685bef052b501bdbe38b"} Jul 24 13:49:34.668381 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:34.668313 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" podStartSLOduration=1.572192533 podStartE2EDuration="2.668288796s" totalImagesPullingTime="1.096096263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:49:34.667543129 +0000 UTC m=+668.264107202" watchObservedRunningTime="2026-07-24 13:49:34.668288796 +0000 UTC m=+668.264852854" Jul 24 13:49:39.865101 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:39.865067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:49:43.046301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:43.046254 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:43.046837 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:43.046335 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:43.058940 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:43.058916 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:49:43.701825 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:49:43.701796 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv" Jul 24 13:50:04.222974 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.222941 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x"] Jul 24 13:50:04.226304 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.226281 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.228830 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.228804 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 13:50:04.236710 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.236684 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x"] Jul 24 13:50:04.290926 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.290880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.290926 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.290921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.291141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.290940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.291141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.290966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.291141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.291080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.291141 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.291125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpjz\" (UniqueName: \"kubernetes.io/projected/fabaf22d-f85d-433b-8b87-3c2d701d85b7-kube-api-access-4rpjz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392012 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.391978 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392199 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpjz\" (UniqueName: \"kubernetes.io/projected/fabaf22d-f85d-433b-8b87-3c2d701d85b7-kube-api-access-4rpjz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392199 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392199 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392084 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392199 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392199 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392601 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392739 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.392739 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.392693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.394411 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.394389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fabaf22d-f85d-433b-8b87-3c2d701d85b7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.394544 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.394528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fabaf22d-f85d-433b-8b87-3c2d701d85b7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.399608 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.399565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpjz\" (UniqueName: \"kubernetes.io/projected/fabaf22d-f85d-433b-8b87-3c2d701d85b7-kube-api-access-4rpjz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4k54x\" (UID: \"fabaf22d-f85d-433b-8b87-3c2d701d85b7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.412816 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.412782 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:50:04.540197 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.540105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:04.680739 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.680702 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x"] Jul 24 13:50:04.683429 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:50:04.683400 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabaf22d_f85d_433b_8b87_3c2d701d85b7.slice/crio-02589b143f3bd08fb7612cf0bcd9fe33eb442f05b9ce18a4fd82e88d38b7b6de WatchSource:0}: Error finding container 02589b143f3bd08fb7612cf0bcd9fe33eb442f05b9ce18a4fd82e88d38b7b6de: Status 404 returned error can't find the container with id 02589b143f3bd08fb7612cf0bcd9fe33eb442f05b9ce18a4fd82e88d38b7b6de Jul 24 13:50:04.788197 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:04.788164 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" event={"ID":"fabaf22d-f85d-433b-8b87-3c2d701d85b7","Type":"ContainerStarted","Data":"02589b143f3bd08fb7612cf0bcd9fe33eb442f05b9ce18a4fd82e88d38b7b6de"} Jul 24 13:50:05.795134 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:05.795096 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" event={"ID":"fabaf22d-f85d-433b-8b87-3c2d701d85b7","Type":"ContainerStarted","Data":"813df64a0c9faa376791b898fc748f3739533492314f3ca5e95b7640657b21de"} Jul 24 13:50:05.812763 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:05.812687 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" podStartSLOduration=1.5987138330000001 podStartE2EDuration="1.812662376s" totalImagesPullingTime="213.948543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:50:05.811364864 +0000 UTC m=+699.407928920" watchObservedRunningTime="2026-07-24 13:50:05.812662376 +0000 UTC m=+699.409226432" Jul 24 13:50:07.638916 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:07.638885 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:50:14.540990 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:14.540941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:14.541631 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:14.541043 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:14.553743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:14.553716 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:14.847618 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:14.847505 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4k54x" Jul 24 13:50:17.066672 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:17.066631 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:50:21.183237 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:21.183148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:50:21.282673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:21.282641 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:50:28.848688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:50:28.848649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:13.111713 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:13.111666 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:13.212634 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:13.212602 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:34.581429 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:34.581390 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:34.692652 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:34.692620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:35.687523 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.687488 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:51:35.688291 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.688243 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" podUID="396cab13-7532-45d9-a187-78fe0f5c5d1b" containerName="authorino" containerID="cri-o://5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237" gracePeriod=30 Jul 24 13:51:35.737800 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.737765 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:35.832434 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.832402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:51:35.942895 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.942828 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:51:35.974976 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.974945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert\") pod \"396cab13-7532-45d9-a187-78fe0f5c5d1b\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " Jul 24 13:51:35.975264 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.975077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cnkp\" (UniqueName: \"kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp\") pod \"396cab13-7532-45d9-a187-78fe0f5c5d1b\" (UID: \"396cab13-7532-45d9-a187-78fe0f5c5d1b\") " Jul 24 13:51:35.977623 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.977563 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp" (OuterVolumeSpecName: "kube-api-access-2cnkp") pod "396cab13-7532-45d9-a187-78fe0f5c5d1b" (UID: "396cab13-7532-45d9-a187-78fe0f5c5d1b"). InnerVolumeSpecName "kube-api-access-2cnkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:51:35.986055 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:35.986024 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "396cab13-7532-45d9-a187-78fe0f5c5d1b" (UID: "396cab13-7532-45d9-a187-78fe0f5c5d1b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:51:36.076089 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.076054 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cnkp\" (UniqueName: \"kubernetes.io/projected/396cab13-7532-45d9-a187-78fe0f5c5d1b-kube-api-access-2cnkp\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:51:36.076089 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.076082 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/396cab13-7532-45d9-a187-78fe0f5c5d1b-tls-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:51:36.205120 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.205029 2581 generic.go:356] "Generic (PLEG): container finished" podID="396cab13-7532-45d9-a187-78fe0f5c5d1b" containerID="5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237" exitCode=0 Jul 24 13:51:36.205120 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.205075 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" Jul 24 13:51:36.205120 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.205073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" event={"ID":"396cab13-7532-45d9-a187-78fe0f5c5d1b","Type":"ContainerDied","Data":"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237"} Jul 24 13:51:36.205369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.205122 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cfc6c5fb5-pqcpn" event={"ID":"396cab13-7532-45d9-a187-78fe0f5c5d1b","Type":"ContainerDied","Data":"554611144eb53a2c927b11dbd560b6ef0f13e2345e4186d3bca2c02c3546d599"} Jul 24 13:51:36.205369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.205145 2581 scope.go:117] "RemoveContainer" containerID="5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237" Jul 24 13:51:36.214851 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.214826 2581 scope.go:117] "RemoveContainer" containerID="5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237" Jul 24 13:51:36.215147 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:51:36.215127 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237\": container with ID starting with 5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237 not found: ID does not exist" containerID="5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237" Jul 24 13:51:36.215216 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.215158 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237"} err="failed to get container status \"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237\": rpc error: code = NotFound desc = could not find container \"5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237\": container with ID starting with 5f3581b07b5b11524322aca16304206385907031f33a6746061025b7f8176237 not found: ID does not exist" Jul 24 13:51:36.232808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.232779 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:51:36.235681 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.235654 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cfc6c5fb5-pqcpn"] Jul 24 13:51:36.916301 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:51:36.916267 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396cab13-7532-45d9-a187-78fe0f5c5d1b" path="/var/lib/kubelet/pods/396cab13-7532-45d9-a187-78fe0f5c5d1b/volumes" Jul 24 13:52:22.508578 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.508537 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:52:22.509143 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.508772 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84b69666bc-4r4tr" podUID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" containerName="manager" containerID="cri-o://160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58" gracePeriod=30 Jul 24 13:52:22.765981 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.765900 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:52:22.788414 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.788387 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rbd8\" (UniqueName: \"kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8\") pod \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " Jul 24 13:52:22.791750 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.791692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8" (OuterVolumeSpecName: "kube-api-access-5rbd8") pod "b880b3b6-ab06-4d50-aa91-70e4c3e53770" (UID: "b880b3b6-ab06-4d50-aa91-70e4c3e53770"). InnerVolumeSpecName "kube-api-access-5rbd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:52:22.889107 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.889059 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert\") pod \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\" (UID: \"b880b3b6-ab06-4d50-aa91-70e4c3e53770\") " Jul 24 13:52:22.889335 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.889317 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbd8\" (UniqueName: \"kubernetes.io/projected/b880b3b6-ab06-4d50-aa91-70e4c3e53770-kube-api-access-5rbd8\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:52:22.891103 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.891075 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b880b3b6-ab06-4d50-aa91-70e4c3e53770" (UID: "b880b3b6-ab06-4d50-aa91-70e4c3e53770"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:52:22.990193 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:22.990159 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b880b3b6-ab06-4d50-aa91-70e4c3e53770-webhook-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:52:23.425543 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.425505 2581 generic.go:356] "Generic (PLEG): container finished" podID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" containerID="160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58" exitCode=0 Jul 24 13:52:23.425743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.425557 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-4r4tr" Jul 24 13:52:23.425743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.425571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-4r4tr" event={"ID":"b880b3b6-ab06-4d50-aa91-70e4c3e53770","Type":"ContainerDied","Data":"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58"} Jul 24 13:52:23.425743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.425644 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-4r4tr" event={"ID":"b880b3b6-ab06-4d50-aa91-70e4c3e53770","Type":"ContainerDied","Data":"1d247aa36c1f979a3753375bba27d8a157be5041497e804cd5d5ca3edf740ac4"} Jul 24 13:52:23.425743 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.425677 2581 scope.go:117] "RemoveContainer" containerID="160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58" Jul 24 13:52:23.434808 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.434789 2581 scope.go:117] "RemoveContainer" containerID="160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58" Jul 24 13:52:23.435091 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:52:23.435074 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58\": container with ID starting with 160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58 not found: ID does not exist" containerID="160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58" Jul 24 13:52:23.435131 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.435102 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58"} err="failed to get container status \"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58\": rpc error: code = NotFound desc = could not find container \"160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58\": container with ID starting with 160d8e98ec10c2d7001cde68e961025be66f05e5f3765e0a6324b480fa88ea58 not found: ID does not exist" Jul 24 13:52:23.442268 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.442220 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:52:23.445103 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:23.445079 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84b69666bc-4r4tr"] Jul 24 13:52:24.921782 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:24.921740 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" path="/var/lib/kubelet/pods/b880b3b6-ab06-4d50-aa91-70e4c3e53770/volumes" Jul 24 13:52:52.101572 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:52.101536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:52:52.202987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:52:52.202953 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:53:24.332569 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332491 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332879 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" containerName="manager" Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332891 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" containerName="manager" Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332904 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396cab13-7532-45d9-a187-78fe0f5c5d1b" containerName="authorino" Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332910 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="396cab13-7532-45d9-a187-78fe0f5c5d1b" containerName="authorino" Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332961 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="396cab13-7532-45d9-a187-78fe0f5c5d1b" containerName="authorino" Jul 24 13:53:24.333092 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.332970 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b880b3b6-ab06-4d50-aa91-70e4c3e53770" containerName="manager" Jul 24 13:53:24.335916 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.335895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.338352 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.338330 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 13:53:24.339048 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.339026 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 13:53:24.339048 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.339048 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8s96w\"" Jul 24 13:53:24.346868 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.346549 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:24.394665 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.394627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8257\" (UniqueName: \"kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.394856 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.394769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.495676 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.495635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.495888 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.495702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8257\" (UniqueName: \"kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.498043 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.498019 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.504016 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.503989 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8257\" (UniqueName: \"kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257\") pod \"maas-controller-84b69666bc-ll795\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.665952 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.665922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:24.806951 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:24.806883 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:24.809261 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:53:24.809236 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d119a6_b986_4bad_b045_a45550850ed4.slice/crio-bc0ce33c18635792d5b7c0215bf99c81314340e429fffe896f52cf4ac560025c WatchSource:0}: Error finding container bc0ce33c18635792d5b7c0215bf99c81314340e429fffe896f52cf4ac560025c: Status 404 returned error can't find the container with id bc0ce33c18635792d5b7c0215bf99c81314340e429fffe896f52cf4ac560025c Jul 24 13:53:25.710279 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:25.710237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-ll795" event={"ID":"e2d119a6-b986-4bad-b045-a45550850ed4","Type":"ContainerStarted","Data":"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2"} Jul 24 13:53:25.710719 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:25.710324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-ll795" event={"ID":"e2d119a6-b986-4bad-b045-a45550850ed4","Type":"ContainerStarted","Data":"bc0ce33c18635792d5b7c0215bf99c81314340e429fffe896f52cf4ac560025c"} Jul 24 13:53:25.710719 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:25.710514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:25.729510 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:25.729448 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b69666bc-ll795" podStartSLOduration=1.209819723 podStartE2EDuration="1.729430067s" totalImagesPullingTime="519.610344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:53:25.727901054 +0000 UTC m=+899.324465191" watchObservedRunningTime="2026-07-24 13:53:25.729430067 +0000 UTC m=+899.325994123" Jul 24 13:53:36.718040 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:36.718005 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:37.105114 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:37.105032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:53:37.204942 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:37.204909 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:53:41.286329 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.286287 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:53:41.381004 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.380967 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:41.381240 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.381216 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84b69666bc-ll795" podUID="e2d119a6-b986-4bad-b045-a45550850ed4" containerName="manager" containerID="cri-o://9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2" gracePeriod=30 Jul 24 13:53:41.627567 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.627536 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:41.737966 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.737927 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert\") pod \"e2d119a6-b986-4bad-b045-a45550850ed4\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " Jul 24 13:53:41.738169 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.738034 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8257\" (UniqueName: \"kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257\") pod \"e2d119a6-b986-4bad-b045-a45550850ed4\" (UID: \"e2d119a6-b986-4bad-b045-a45550850ed4\") " Jul 24 13:53:41.740161 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.740134 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e2d119a6-b986-4bad-b045-a45550850ed4" (UID: "e2d119a6-b986-4bad-b045-a45550850ed4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:53:41.740252 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.740226 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257" (OuterVolumeSpecName: "kube-api-access-s8257") pod "e2d119a6-b986-4bad-b045-a45550850ed4" (UID: "e2d119a6-b986-4bad-b045-a45550850ed4"). InnerVolumeSpecName "kube-api-access-s8257". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:53:41.790203 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.790168 2581 generic.go:356] "Generic (PLEG): container finished" podID="e2d119a6-b986-4bad-b045-a45550850ed4" containerID="9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2" exitCode=0 Jul 24 13:53:41.790377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.790212 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-ll795" Jul 24 13:53:41.790377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.790260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-ll795" event={"ID":"e2d119a6-b986-4bad-b045-a45550850ed4","Type":"ContainerDied","Data":"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2"} Jul 24 13:53:41.790377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.790316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-ll795" event={"ID":"e2d119a6-b986-4bad-b045-a45550850ed4","Type":"ContainerDied","Data":"bc0ce33c18635792d5b7c0215bf99c81314340e429fffe896f52cf4ac560025c"} Jul 24 13:53:41.790377 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.790340 2581 scope.go:117] "RemoveContainer" containerID="9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2" Jul 24 13:53:41.799205 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.799186 2581 scope.go:117] "RemoveContainer" containerID="9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2" Jul 24 13:53:41.799499 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:53:41.799471 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2\": container with ID starting with 9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2 not found: ID does not exist" containerID="9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2" Jul 24 13:53:41.799545 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.799508 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2"} err="failed to get container status \"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2\": rpc error: code = NotFound desc = could not find container \"9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2\": container with ID starting with 9f2ea6177de66129216275f3599e426528978357fa2cf8ab0ad747a6183869d2 not found: ID does not exist" Jul 24 13:53:41.811789 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.811711 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:41.815158 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.815125 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84b69666bc-ll795"] Jul 24 13:53:41.838887 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.838855 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d119a6-b986-4bad-b045-a45550850ed4-webhook-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:53:41.838887 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:41.838887 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8257\" (UniqueName: \"kubernetes.io/projected/e2d119a6-b986-4bad-b045-a45550850ed4-kube-api-access-s8257\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:53:42.607096 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.607059 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b69666bc-7wcrr"] Jul 24 13:53:42.607499 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.607425 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d119a6-b986-4bad-b045-a45550850ed4" containerName="manager" Jul 24 13:53:42.607499 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.607435 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d119a6-b986-4bad-b045-a45550850ed4" containerName="manager" Jul 24 13:53:42.607499 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.607497 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d119a6-b986-4bad-b045-a45550850ed4" containerName="manager" Jul 24 13:53:42.611815 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.611791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.615263 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.615235 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8s96w\"" Jul 24 13:53:42.615404 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.615235 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 13:53:42.615404 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.615239 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 13:53:42.624099 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.624066 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-7wcrr"] Jul 24 13:53:42.646335 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.646294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce567867-5598-469e-b323-85cd68ff2e53-webhook-cert\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.646522 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.646357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57vg\" (UniqueName: \"kubernetes.io/projected/ce567867-5598-469e-b323-85cd68ff2e53-kube-api-access-h57vg\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.747428 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.747390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce567867-5598-469e-b323-85cd68ff2e53-webhook-cert\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.747428 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.747435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57vg\" (UniqueName: \"kubernetes.io/projected/ce567867-5598-469e-b323-85cd68ff2e53-kube-api-access-h57vg\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.749965 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.749939 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce567867-5598-469e-b323-85cd68ff2e53-webhook-cert\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.756877 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.756848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57vg\" (UniqueName: \"kubernetes.io/projected/ce567867-5598-469e-b323-85cd68ff2e53-kube-api-access-h57vg\") pod \"maas-controller-84b69666bc-7wcrr\" (UID: \"ce567867-5598-469e-b323-85cd68ff2e53\") " pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:42.916484 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.916453 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d119a6-b986-4bad-b045-a45550850ed4" path="/var/lib/kubelet/pods/e2d119a6-b986-4bad-b045-a45550850ed4/volumes" Jul 24 13:53:42.922653 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:42.922617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:43.067568 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.067542 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b69666bc-7wcrr"] Jul 24 13:53:43.070170 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:53:43.070141 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce567867_5598_469e_b323_85cd68ff2e53.slice/crio-7bfc86a09b4976476f978f85d57d0c8928fb40e1733e426d3aade9d495828059 WatchSource:0}: Error finding container 7bfc86a09b4976476f978f85d57d0c8928fb40e1733e426d3aade9d495828059: Status 404 returned error can't find the container with id 7bfc86a09b4976476f978f85d57d0c8928fb40e1733e426d3aade9d495828059 Jul 24 13:53:43.256987 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.256908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:53:43.809897 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.809857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-7wcrr" event={"ID":"ce567867-5598-469e-b323-85cd68ff2e53","Type":"ContainerStarted","Data":"6ac7fd4a832c88da9c9943f0fee6a4d055ad2c362fe1e6ff78370d06f227a6ee"} Jul 24 13:53:43.810688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.809948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b69666bc-7wcrr" event={"ID":"ce567867-5598-469e-b323-85cd68ff2e53","Type":"ContainerStarted","Data":"7bfc86a09b4976476f978f85d57d0c8928fb40e1733e426d3aade9d495828059"} Jul 24 13:53:43.810688 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.809989 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:53:43.829297 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:43.829241 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b69666bc-7wcrr" podStartSLOduration=1.285879527 podStartE2EDuration="1.829221517s" totalImagesPullingTime="543.34199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:53:43.827626709 +0000 UTC m=+917.424190764" watchObservedRunningTime="2026-07-24 13:53:43.829221517 +0000 UTC m=+917.425785571" Jul 24 13:53:54.817287 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:53:54.817259 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b69666bc-7wcrr" Jul 24 13:54:16.491952 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:16.491912 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:16.587401 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:16.587363 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:16.720673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:16.720643 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:22.590356 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:22.590315 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:22.696160 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:22.696121 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:22.794195 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:22.794154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:23.016125 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:23.016086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:28.842330 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:28.842295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:54:28.951870 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:54:28.951822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:55:43.475767 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:55:43.475730 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:55:52.581460 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:55:52.581422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:56:18.960293 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:56:18.960214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:58:31.412336 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:31.412299 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:58:31.412926 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:31.412512 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" podUID="d7c9608a-efdd-450c-9600-61a77218dddd" containerName="manager" containerID="cri-o://042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4" gracePeriod=10 Jul 24 13:58:34.972716 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:34.972690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:58:35.045277 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.045197 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw2lm\" (UniqueName: \"kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm\") pod \"d7c9608a-efdd-450c-9600-61a77218dddd\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " Jul 24 13:58:35.045277 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.045242 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume\") pod \"d7c9608a-efdd-450c-9600-61a77218dddd\" (UID: \"d7c9608a-efdd-450c-9600-61a77218dddd\") " Jul 24 13:58:35.045696 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.045671 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d7c9608a-efdd-450c-9600-61a77218dddd" (UID: "d7c9608a-efdd-450c-9600-61a77218dddd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:58:35.047329 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.047305 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm" (OuterVolumeSpecName: "kube-api-access-zw2lm") pod "d7c9608a-efdd-450c-9600-61a77218dddd" (UID: "d7c9608a-efdd-450c-9600-61a77218dddd"). InnerVolumeSpecName "kube-api-access-zw2lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:58:35.138885 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.138851 2581 generic.go:356] "Generic (PLEG): container finished" podID="d7c9608a-efdd-450c-9600-61a77218dddd" containerID="042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4" exitCode=0 Jul 24 13:58:35.139053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.138898 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" Jul 24 13:58:35.139053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.138915 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" event={"ID":"d7c9608a-efdd-450c-9600-61a77218dddd","Type":"ContainerDied","Data":"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4"} Jul 24 13:58:35.139053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.139005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z" event={"ID":"d7c9608a-efdd-450c-9600-61a77218dddd","Type":"ContainerDied","Data":"626fb46f631db5399d84060d789086a2a12917b2142e260dc783c198432f815f"} Jul 24 13:58:35.139053 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.139032 2581 scope.go:117] "RemoveContainer" containerID="042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4" Jul 24 13:58:35.146369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.146346 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw2lm\" (UniqueName: \"kubernetes.io/projected/d7c9608a-efdd-450c-9600-61a77218dddd-kube-api-access-zw2lm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:58:35.146369 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.146369 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c9608a-efdd-450c-9600-61a77218dddd-extensions-socket-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 13:58:35.149118 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.149099 2581 scope.go:117] "RemoveContainer" containerID="042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4" Jul 24 13:58:35.149429 ip-10-0-143-193 kubenswrapper[2581]: E0724 13:58:35.149409 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4\": container with ID starting with 042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4 not found: ID does not exist" containerID="042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4" Jul 24 13:58:35.149484 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.149440 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4"} err="failed to get container status \"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4\": rpc error: code = NotFound desc = could not find container \"042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4\": container with ID starting with 042ef58fb97fd76a9f74d7e5db73de6d81bb1255636ed6d02e81273d1c2630c4 not found: ID does not exist" Jul 24 13:58:35.160976 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.160944 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:58:35.164931 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:35.164900 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wf48z"] Jul 24 13:58:36.916414 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:58:36.916387 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c9608a-efdd-450c-9600-61a77218dddd" path="/var/lib/kubelet/pods/d7c9608a-efdd-450c-9600-61a77218dddd/volumes" Jul 24 13:59:02.665413 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:02.665364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:59:21.076274 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:21.076199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 13:59:32.629216 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.629176 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj"] Jul 24 13:59:32.629628 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.629599 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c9608a-efdd-450c-9600-61a77218dddd" containerName="manager" Jul 24 13:59:32.629628 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.629611 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c9608a-efdd-450c-9600-61a77218dddd" containerName="manager" Jul 24 13:59:32.629705 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.629686 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c9608a-efdd-450c-9600-61a77218dddd" containerName="manager" Jul 24 13:59:32.632944 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.632922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.635955 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.635936 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fsd7k\"" Jul 24 13:59:32.657451 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.657415 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj"] Jul 24 13:59:32.733484 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.733435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba6fca0-967f-4396-9425-85d96d440879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.733484 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.733488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84p8\" (UniqueName: \"kubernetes.io/projected/bba6fca0-967f-4396-9425-85d96d440879-kube-api-access-j84p8\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.834313 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.834276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84p8\" (UniqueName: \"kubernetes.io/projected/bba6fca0-967f-4396-9425-85d96d440879-kube-api-access-j84p8\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.834628 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.834366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba6fca0-967f-4396-9425-85d96d440879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.834710 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.834694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bba6fca0-967f-4396-9425-85d96d440879-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.845299 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.845272 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84p8\" (UniqueName: \"kubernetes.io/projected/bba6fca0-967f-4396-9425-85d96d440879-kube-api-access-j84p8\") pod \"kuadrant-operator-controller-manager-86878c769f-8s4jj\" (UID: \"bba6fca0-967f-4396-9425-85d96d440879\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:32.948673 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:32.948638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:33.106946 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:33.106914 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj"] Jul 24 13:59:33.111545 ip-10-0-143-193 kubenswrapper[2581]: W0724 13:59:33.111512 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba6fca0_967f_4396_9425_85d96d440879.slice/crio-93b2797311ed6e54499127782246e150105ab3dffbbcc3a7c7590799a45440c7 WatchSource:0}: Error finding container 93b2797311ed6e54499127782246e150105ab3dffbbcc3a7c7590799a45440c7: Status 404 returned error can't find the container with id 93b2797311ed6e54499127782246e150105ab3dffbbcc3a7c7590799a45440c7 Jul 24 13:59:33.412414 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:33.412321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" event={"ID":"bba6fca0-967f-4396-9425-85d96d440879","Type":"ContainerStarted","Data":"eacb39d21c6cc1218c3905022ed615d9793bf42daba9188047a38becad3a92a2"} Jul 24 13:59:33.412414 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:33.412390 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" event={"ID":"bba6fca0-967f-4396-9425-85d96d440879","Type":"ContainerStarted","Data":"93b2797311ed6e54499127782246e150105ab3dffbbcc3a7c7590799a45440c7"} Jul 24 13:59:33.412687 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:33.412439 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 13:59:33.435865 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:33.435812 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" podStartSLOduration=1.435793524 podStartE2EDuration="1.435793524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:59:33.433456428 +0000 UTC m=+1267.030020502" watchObservedRunningTime="2026-07-24 13:59:33.435793524 +0000 UTC m=+1267.032357579" Jul 24 13:59:44.419477 ip-10-0-143-193 kubenswrapper[2581]: I0724 13:59:44.419448 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8s4jj" Jul 24 14:00:00.146079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.146041 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf"] Jul 24 14:00:00.149330 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.149312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" Jul 24 14:00:00.151513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.151490 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:00:00.156747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.156719 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf"] Jul 24 14:00:00.160087 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.160064 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" Jul 24 14:00:00.299313 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.299283 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf"] Jul 24 14:00:00.301414 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:00:00.301387 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7556b2f0_ee9d_428a_96a1_027fcf18e8a3.slice/crio-aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37 WatchSource:0}: Error finding container aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37: Status 404 returned error can't find the container with id aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37 Jul 24 14:00:00.303748 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.303728 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:00:00.534953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:00.534916 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" event={"ID":"7556b2f0-ee9d-428a-96a1-027fcf18e8a3","Type":"ContainerStarted","Data":"aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37"} Jul 24 14:00:04.556390 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:04.556290 2581 generic.go:356] "Generic (PLEG): container finished" podID="7556b2f0-ee9d-428a-96a1-027fcf18e8a3" containerID="93857220e84d6f4bc20768b5d987f37d8c7467768170ef1f02a361c5f22a5795" exitCode=0 Jul 24 14:00:04.556390 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:04.556338 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" event={"ID":"7556b2f0-ee9d-428a-96a1-027fcf18e8a3","Type":"ContainerDied","Data":"93857220e84d6f4bc20768b5d987f37d8c7467768170ef1f02a361c5f22a5795"} Jul 24 14:00:05.700408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:05.700373 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" Jul 24 14:00:06.568368 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:06.568337 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" Jul 24 14:00:06.568558 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:06.568343 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748360-bj5xf" event={"ID":"7556b2f0-ee9d-428a-96a1-027fcf18e8a3","Type":"ContainerDied","Data":"aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37"} Jul 24 14:00:06.568558 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:00:06.568468 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff1149c09133128d85ed17bb5b77832c0874394ea5e26b0ea5c610071089b37" Jul 24 14:01:25.528227 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:01:25.528144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:01:59.736832 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:01:59.736798 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:02:20.628324 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.628240 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:02:20.628724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.628671 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7556b2f0-ee9d-428a-96a1-027fcf18e8a3" containerName="cleanup" Jul 24 14:02:20.628724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.628686 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7556b2f0-ee9d-428a-96a1-027fcf18e8a3" containerName="cleanup" Jul 24 14:02:20.628796 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.628753 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7556b2f0-ee9d-428a-96a1-027fcf18e8a3" containerName="cleanup" Jul 24 14:02:20.631693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.631674 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.633934 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.633910 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2cf2532c-kserve-self-signed-certs\"" Jul 24 14:02:20.647356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.647317 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:02:20.726025 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.725990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:02:20.748090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.748309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nb6b\" (UniqueName: \"kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.748309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.748309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748217 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.748309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.748309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.748261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849787 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849787 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849787 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nb6b\" (UniqueName: \"kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849787 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849677 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.849993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.849942 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.850051 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.850003 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.850109 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.850095 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.851948 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.851924 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.852161 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.852140 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.857559 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.857537 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nb6b\" (UniqueName: \"kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b\") pod \"e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:20.941136 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:20.941093 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:21.079399 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:21.079373 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:02:21.082024 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:02:21.081994 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aa13aef_f1d3_4bee_9fc3_329dfbe9cccf.slice/crio-977f0f82d19589540ff5eaf58fe70f93eb63ace26645564b51908b98ce61dad7 WatchSource:0}: Error finding container 977f0f82d19589540ff5eaf58fe70f93eb63ace26645564b51908b98ce61dad7: Status 404 returned error can't find the container with id 977f0f82d19589540ff5eaf58fe70f93eb63ace26645564b51908b98ce61dad7 Jul 24 14:02:21.216895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:21.216807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" event={"ID":"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf","Type":"ContainerStarted","Data":"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e"} Jul 24 14:02:21.216895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:21.216852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" event={"ID":"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf","Type":"ContainerStarted","Data":"977f0f82d19589540ff5eaf58fe70f93eb63ace26645564b51908b98ce61dad7"} Jul 24 14:02:21.234665 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:21.234615 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" podStartSLOduration=1.234597798 podStartE2EDuration="1.234597798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:21.233619754 +0000 UTC m=+1434.830183810" watchObservedRunningTime="2026-07-24 14:02:21.234597798 +0000 UTC m=+1434.831161848" Jul 24 14:02:30.942020 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:30.941977 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:30.942523 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:30.942074 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:30.955060 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:30.955031 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:31.276776 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:31.276678 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:02:32.123323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.123283 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:02:32.127374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.127349 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.130086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.130060 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2cf2532c-kserve-self-signed-certs\"" Jul 24 14:02:32.136464 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.136438 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:02:32.177807 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.177756 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:02:32.246775 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.246775 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.246999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.246999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqfq\" (UniqueName: \"kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.246999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.246999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.246966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347601 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347752 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.347961 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.347817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqfq\" (UniqueName: \"kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.348043 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.348024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.348141 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.348111 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.348359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.348342 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.350273 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.350246 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.350273 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.350269 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.356426 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.356394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqfq\" (UniqueName: \"kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq\") pod \"e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.438792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.438749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:32.582981 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:32.582945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:02:32.585995 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:02:32.585969 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e019488_9dd2_4d6a_b0a9_1da94e8248fc.slice/crio-3ff4ec254737994b135ad7282dea06463d1cf9c9419f69146c12bf11100827eb WatchSource:0}: Error finding container 3ff4ec254737994b135ad7282dea06463d1cf9c9419f69146c12bf11100827eb: Status 404 returned error can't find the container with id 3ff4ec254737994b135ad7282dea06463d1cf9c9419f69146c12bf11100827eb Jul 24 14:02:33.278872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:33.278836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" event={"ID":"2e019488-9dd2-4d6a-b0a9-1da94e8248fc","Type":"ContainerStarted","Data":"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86"} Jul 24 14:02:33.279282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:33.278882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" event={"ID":"2e019488-9dd2-4d6a-b0a9-1da94e8248fc","Type":"ContainerStarted","Data":"3ff4ec254737994b135ad7282dea06463d1cf9c9419f69146c12bf11100827eb"} Jul 24 14:02:33.296979 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:33.296924 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" podStartSLOduration=1.296907496 podStartE2EDuration="1.296907496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:33.295930806 +0000 UTC m=+1446.892494861" watchObservedRunningTime="2026-07-24 14:02:33.296907496 +0000 UTC m=+1446.893471551" Jul 24 14:02:42.439649 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:42.439576 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:42.439649 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:42.439659 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:42.452569 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:42.452542 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:43.337633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:43.337602 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:02:50.367347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:50.367305 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:02:51.270938 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:51.270883 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:02:52.019320 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:52.019280 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:02:52.040678 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:02:52.040647 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:03:05.390562 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.390521 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" podUID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" containerName="main" containerID="cri-o://4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e" gracePeriod=45 Jul 24 14:03:05.654271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.654197 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:03:05.713768 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713739 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.713768 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713769 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.714047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713808 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.714047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713833 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.714047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713883 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.714047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.713919 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nb6b\" (UniqueName: \"kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b\") pod \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\" (UID: \"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf\") " Jul 24 14:03:05.714268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.714093 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home" (OuterVolumeSpecName: "home") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:05.714268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.714167 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-home\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:05.714373 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.714293 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache" (OuterVolumeSpecName: "model-cache") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:05.714437 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.714361 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:05.716337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.716304 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b" (OuterVolumeSpecName: "kube-api-access-2nb6b") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "kube-api-access-2nb6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:03:05.716476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.716372 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:03:05.716476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.716432 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm" (OuterVolumeSpecName: "dshm") pod "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" (UID: "3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:05.815350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.815296 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tls-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:05.815350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.815343 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-model-cache\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:05.815350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.815352 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-tmp-dir\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:05.815350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.815360 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nb6b\" (UniqueName: \"kubernetes.io/projected/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-kube-api-access-2nb6b\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:05.815350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:05.815370 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf-dshm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:06.437214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.437180 2581 generic.go:356] "Generic (PLEG): container finished" podID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" containerID="4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e" exitCode=0 Jul 24 14:03:06.437710 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.437227 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" Jul 24 14:03:06.437710 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.437265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" event={"ID":"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf","Type":"ContainerDied","Data":"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e"} Jul 24 14:03:06.437710 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.437321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n" event={"ID":"3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf","Type":"ContainerDied","Data":"977f0f82d19589540ff5eaf58fe70f93eb63ace26645564b51908b98ce61dad7"} Jul 24 14:03:06.437710 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.437345 2581 scope.go:117] "RemoveContainer" containerID="4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e" Jul 24 14:03:06.446992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.446971 2581 scope.go:117] "RemoveContainer" containerID="4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e" Jul 24 14:03:06.447320 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:03:06.447301 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e\": container with ID starting with 4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e not found: ID does not exist" containerID="4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e" Jul 24 14:03:06.447383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.447329 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e"} err="failed to get container status \"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e\": rpc error: code = NotFound desc = could not find container \"4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e\": container with ID starting with 4e2a9037cbb56e57522c66dc784fe75a1ed0a485285760f3bd527ec3094dcc6e not found: ID does not exist" Jul 24 14:03:06.459247 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.459207 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:03:06.461135 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.461107 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2cf2532c-kserve-6fdddf9b75-qh56n"] Jul 24 14:03:06.916594 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:06.916556 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" path="/var/lib/kubelet/pods/3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf/volumes" Jul 24 14:03:07.040436 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.040388 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" podUID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" containerName="main" containerID="cri-o://057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86" gracePeriod=45 Jul 24 14:03:07.305131 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.305098 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:03:07.329926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.329891 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.329995 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330039 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330085 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npqfq\" (UniqueName: \"kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330121 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330168 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache\") pod \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\" (UID: \"2e019488-9dd2-4d6a-b0a9-1da94e8248fc\") " Jul 24 14:03:07.330404 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330368 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:07.330651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330624 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:07.330744 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.330635 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home" (OuterVolumeSpecName: "home") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:07.332191 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.332161 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm" (OuterVolumeSpecName: "dshm") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:03:07.332630 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.332602 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq" (OuterVolumeSpecName: "kube-api-access-npqfq") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "kube-api-access-npqfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:03:07.332630 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.332615 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e019488-9dd2-4d6a-b0a9-1da94e8248fc" (UID: "2e019488-9dd2-4d6a-b0a9-1da94e8248fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:03:07.431381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431339 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-model-cache\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.431381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431373 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tls-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.431381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431383 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-dshm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.431381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431391 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-tmp-dir\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.431704 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431400 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqfq\" (UniqueName: \"kubernetes.io/projected/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-kube-api-access-npqfq\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.431704 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.431409 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e019488-9dd2-4d6a-b0a9-1da94e8248fc-home\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:03:07.444777 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.444745 2581 generic.go:356] "Generic (PLEG): container finished" podID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" containerID="057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86" exitCode=0 Jul 24 14:03:07.445211 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.444792 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" Jul 24 14:03:07.445211 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.444790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" event={"ID":"2e019488-9dd2-4d6a-b0a9-1da94e8248fc","Type":"ContainerDied","Data":"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86"} Jul 24 14:03:07.445211 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.444928 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b" event={"ID":"2e019488-9dd2-4d6a-b0a9-1da94e8248fc","Type":"ContainerDied","Data":"3ff4ec254737994b135ad7282dea06463d1cf9c9419f69146c12bf11100827eb"} Jul 24 14:03:07.445211 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.444959 2581 scope.go:117] "RemoveContainer" containerID="057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86" Jul 24 14:03:07.454129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.454101 2581 scope.go:117] "RemoveContainer" containerID="057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86" Jul 24 14:03:07.454442 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:03:07.454418 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86\": container with ID starting with 057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86 not found: ID does not exist" containerID="057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86" Jul 24 14:03:07.454514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.454455 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86"} err="failed to get container status \"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86\": rpc error: code = NotFound desc = could not find container \"057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86\": container with ID starting with 057bec223614ed56bad0e53dd0b6e0e43eb76ef936ea6f5df814dc970cad3a86 not found: ID does not exist" Jul 24 14:03:07.466644 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.466552 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:03:07.471417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:07.471386 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2cf2532c-kserve-6b6f8787ff-cxz4b"] Jul 24 14:03:08.916122 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:03:08.916089 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" path="/var/lib/kubelet/pods/2e019488-9dd2-4d6a-b0a9-1da94e8248fc/volumes" Jul 24 14:06:10.644552 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:10.644519 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:38.524657 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:38.524616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:41.729282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:41.729248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:41.821690 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:41.821655 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:42.184041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:42.184004 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:48.236817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.236734 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:48.302549 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.302518 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:48.497138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.497055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:48.818253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818166 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:06:48.818740 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818723 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" containerName="main" Jul 24 14:06:48.818797 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818743 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" containerName="main" Jul 24 14:06:48.818797 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818770 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" containerName="main" Jul 24 14:06:48.818797 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818778 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" containerName="main" Jul 24 14:06:48.818895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818856 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e019488-9dd2-4d6a-b0a9-1da94e8248fc" containerName="main" Jul 24 14:06:48.818895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.818872 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa13aef-f1d3-4bee-9fc3-329dfbe9cccf" containerName="main" Jul 24 14:06:48.824676 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.823070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.827337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.827307 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 14:06:48.827541 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.827367 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-35e2fd78-openshift-default-dockercfg-wlj8x\"" Jul 24 14:06:48.839130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.839092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:06:48.845301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.845260 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:48.856670 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856670 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqblg\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.856923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.857179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.857179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.856960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.957993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.957953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.957993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.957995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958069 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958097 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqblg\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958738 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958908 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.958978 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.958926 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.959036 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.959027 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.960521 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.960503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.960901 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.960879 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.967337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.967295 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:48.967723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:48.967698 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqblg\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg\") pod \"e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:49.080168 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:49.080077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:49.137025 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:49.136988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:49.288731 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:49.288698 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:06:49.291534 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:06:49.291489 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b3998b_ab3d_41d2_a0a5_2daf9135e25f.slice/crio-c0733dde075e4562314553ac205f1cb4121c767465ce2cc5ec2c8b957b5a46d9 WatchSource:0}: Error finding container c0733dde075e4562314553ac205f1cb4121c767465ce2cc5ec2c8b957b5a46d9: Status 404 returned error can't find the container with id c0733dde075e4562314553ac205f1cb4121c767465ce2cc5ec2c8b957b5a46d9 Jul 24 14:06:49.293704 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:49.293685 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:06:49.510868 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:49.510833 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerStarted","Data":"c0733dde075e4562314553ac205f1cb4121c767465ce2cc5ec2c8b957b5a46d9"} Jul 24 14:06:51.976488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:51.976440 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:06:51.976791 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:51.976529 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:06:51.976791 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:51.976562 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:06:52.529317 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:52.529282 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerStarted","Data":"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9"} Jul 24 14:06:52.552554 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:52.552484 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podStartSLOduration=1.870142364 podStartE2EDuration="4.552458638s" totalImagesPullingTime="2.682316274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:52.549886797 +0000 UTC m=+1706.146450853" watchObservedRunningTime="2026-07-24 14:06:52.552458638 +0000 UTC m=+1706.149022695" Jul 24 14:06:53.137197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:53.137158 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:53.142169 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:53.142140 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:53.539539 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:53.539495 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:53.540730 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:53.540706 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:06:53.916694 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:53.916645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.005935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.005898 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.175550 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.175466 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:06:54.179357 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.179335 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.182881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.182859 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:06:54.182881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.182863 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-35e2fd78-serving-cert\"" Jul 24 14:06:54.192226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.192185 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:06:54.198813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.198773 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.207194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.207159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.207400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.207226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.207400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.207342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvp8\" (UniqueName: \"kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.308707 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.308674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.308901 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.308762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvp8\" (UniqueName: \"kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.308901 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.308791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.309017 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:06:54.308903 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-serving-cert: secret "maas-api-e2e-ait-35e2fd78-serving-cert" not found Jul 24 14:06:54.309017 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:06:54.308962 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls podName:272dcb26-94bc-488f-b1d4-f42ce7fcab28 nodeName:}" failed. No retries permitted until 2026-07-24 14:06:54.808944757 +0000 UTC m=+1708.405508791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls") pod "maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" (UID: "272dcb26-94bc-488f-b1d4-f42ce7fcab28") : secret "maas-api-e2e-ait-35e2fd78-serving-cert" not found Jul 24 14:06:54.309494 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.309465 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.319000 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.318957 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvp8\" (UniqueName: \"kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.405336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.405296 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.813431 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.813380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.816136 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.816098 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") pod \"maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:54.855611 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.855526 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.976088 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.976050 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:06:54.977834 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.977803 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:06:54.979912 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.979895 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:54.982640 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.982618 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-35e2fd78\"" Jul 24 14:06:54.983137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.983120 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-35e2fd78-dockercfg-vptn5\"" Jul 24 14:06:54.991809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:54.991781 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:06:55.014649 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.014595 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gz4\" (UniqueName: \"kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.014649 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.014645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.044491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.044460 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:06:55.047731 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.047711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.060272 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.060238 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:06:55.091276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.091195 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:55.116177 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.116070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7qt\" (UniqueName: \"kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.116177 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.116157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gz4\" (UniqueName: \"kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.116478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.116300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.116478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.116367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.117098 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.117078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.127951 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.127922 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gz4\" (UniqueName: \"kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4\") pod \"payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.217881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.217845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.218376 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.217968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7qt\" (UniqueName: \"kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.218770 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.218744 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.227043 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.227009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7qt\" (UniqueName: \"kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt\") pod \"payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.251383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.251350 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:06:55.254501 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:06:55.254471 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272dcb26_94bc_488f_b1d4_f42ce7fcab28.slice/crio-b69963c518a81758955a16bf8deb4171a501bceca11f695c3239b7677993b8f1 WatchSource:0}: Error finding container b69963c518a81758955a16bf8deb4171a501bceca11f695c3239b7677993b8f1: Status 404 returned error can't find the container with id b69963c518a81758955a16bf8deb4171a501bceca11f695c3239b7677993b8f1 Jul 24 14:06:55.292045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.292010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:55.359294 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.359160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:55.517305 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.517165 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:06:55.519737 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:06:55.519693 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21424deb_2cfe_47e9_a441_72c21d7f89eb.slice/crio-c0feaefa48823c72133ff92e86a5fb9a53566350c96dd40672787fb20944f1e2 WatchSource:0}: Error finding container c0feaefa48823c72133ff92e86a5fb9a53566350c96dd40672787fb20944f1e2: Status 404 returned error can't find the container with id c0feaefa48823c72133ff92e86a5fb9a53566350c96dd40672787fb20944f1e2 Jul 24 14:06:55.550347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.550310 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" event={"ID":"21424deb-2cfe-47e9-a441-72c21d7f89eb","Type":"ContainerStarted","Data":"c0feaefa48823c72133ff92e86a5fb9a53566350c96dd40672787fb20944f1e2"} Jul 24 14:06:55.551769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.551739 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" event={"ID":"272dcb26-94bc-488f-b1d4-f42ce7fcab28","Type":"ContainerStarted","Data":"b69963c518a81758955a16bf8deb4171a501bceca11f695c3239b7677993b8f1"} Jul 24 14:06:55.655967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:55.655935 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:06:55.657409 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:06:55.657386 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a4ce20_4dfb_4635_8e91_c016747738e4.slice/crio-6adef82f1ba5e8093d5d1dae7dea85e89c18a39067f669c2f4f70b7818aceb68 WatchSource:0}: Error finding container 6adef82f1ba5e8093d5d1dae7dea85e89c18a39067f669c2f4f70b7818aceb68: Status 404 returned error can't find the container with id 6adef82f1ba5e8093d5d1dae7dea85e89c18a39067f669c2f4f70b7818aceb68 Jul 24 14:06:56.559632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.559369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" event={"ID":"34a4ce20-4dfb-4635-8e91-c016747738e4","Type":"ContainerStarted","Data":"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36"} Jul 24 14:06:56.559632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.559442 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" event={"ID":"34a4ce20-4dfb-4635-8e91-c016747738e4","Type":"ContainerStarted","Data":"6adef82f1ba5e8093d5d1dae7dea85e89c18a39067f669c2f4f70b7818aceb68"} Jul 24 14:06:56.559632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.559549 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:06:56.561303 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.561240 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" event={"ID":"21424deb-2cfe-47e9-a441-72c21d7f89eb","Type":"ContainerStarted","Data":"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c"} Jul 24 14:06:56.561521 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.561391 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:06:56.579973 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:56.579910 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" podStartSLOduration=2.579886802 podStartE2EDuration="2.579886802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:56.578242027 +0000 UTC m=+1710.174806083" watchObservedRunningTime="2026-07-24 14:06:56.579886802 +0000 UTC m=+1710.176450859" Jul 24 14:06:57.569365 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:57.569328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" event={"ID":"272dcb26-94bc-488f-b1d4-f42ce7fcab28","Type":"ContainerStarted","Data":"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f"} Jul 24 14:06:57.569817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:57.569530 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:06:57.592656 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:57.592579 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" podStartSLOduration=1.947983805 podStartE2EDuration="3.59256256s" totalImagesPullingTime="1.644578755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:57.591510644 +0000 UTC m=+1711.188074717" watchObservedRunningTime="2026-07-24 14:06:57.59256256 +0000 UTC m=+1711.189126681" Jul 24 14:06:57.593358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:06:57.593311 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" podStartSLOduration=2.593297713 podStartE2EDuration="2.593297713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:06:56.596785825 +0000 UTC m=+1710.193349881" watchObservedRunningTime="2026-07-24 14:06:57.593297713 +0000 UTC m=+1711.189861768" Jul 24 14:07:03.586500 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:03.586457 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:07:07.572054 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:07.572022 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:07:07.572552 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:07.572114 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:07:08.541771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.541730 2581 patch_prober.go:28] interesting pod/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 14:07:08.542033 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.541799 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 14:07:08.630397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.630364 2581 generic.go:356] "Generic (PLEG): container finished" podID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerID="791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9" exitCode=0 Jul 24 14:07:08.630872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.630414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerDied","Data":"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9"} Jul 24 14:07:08.630872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.630833 2581 scope.go:117] "RemoveContainer" containerID="791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9" Jul 24 14:07:08.632005 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.631979 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:08.632079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.632046 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:08.632115 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:08.632083 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:09.637101 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:09.637068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerStarted","Data":"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5"} Jul 24 14:07:10.137301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:10.137259 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:10.137536 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:10.137514 2581 patch_prober.go:28] interesting pod/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 14:07:10.137659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:10.137618 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 14:07:11.137526 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:11.137479 2581 patch_prober.go:28] interesting pod/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 14:07:11.138045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:11.137553 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 14:07:12.138414 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:12.138375 2581 patch_prober.go:28] interesting pod/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 14:07:12.138911 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:12.138435 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 14:07:13.138192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:13.138155 2581 patch_prober.go:28] interesting pod/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 14:07:13.138463 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:13.138221 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 14:07:14.142342 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:14.142306 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:14.142848 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:14.142797 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:14.143624 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:14.143602 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:30.201694 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.201649 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:07:30.202577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.202514 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" podUID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" containerName="maas-api" containerID="cri-o://e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f" gracePeriod=30 Jul 24 14:07:30.225045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.225007 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:30.256962 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.256929 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:07:30.257198 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.257174 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" podUID="34a4ce20-4dfb-4635-8e91-c016747738e4" containerName="payload-processing" containerID="cri-o://be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36" gracePeriod=30 Jul 24 14:07:30.263346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.263307 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:07:30.263858 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.263827 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" podUID="21424deb-2cfe-47e9-a441-72c21d7f89eb" containerName="payload-pre-processing" containerID="cri-o://fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c" gracePeriod=30 Jul 24 14:07:30.299515 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.299470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:30.355975 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.355934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:30.547226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.547198 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:07:30.571108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.571078 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:07:30.580459 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.580428 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:07:30.656411 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656379 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") pod \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " Jul 24 14:07:30.656620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rvp8\" (UniqueName: \"kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8\") pod \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " Jul 24 14:07:30.656620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656465 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca\") pod \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\" (UID: \"272dcb26-94bc-488f-b1d4-f42ce7fcab28\") " Jul 24 14:07:30.656620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656485 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8gz4\" (UniqueName: \"kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4\") pod \"34a4ce20-4dfb-4635-8e91-c016747738e4\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " Jul 24 14:07:30.656620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656525 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume\") pod \"34a4ce20-4dfb-4635-8e91-c016747738e4\" (UID: \"34a4ce20-4dfb-4635-8e91-c016747738e4\") " Jul 24 14:07:30.656977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656944 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "272dcb26-94bc-488f-b1d4-f42ce7fcab28" (UID: "272dcb26-94bc-488f-b1d4-f42ce7fcab28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:07:30.656977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.656961 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34a4ce20-4dfb-4635-8e91-c016747738e4" (UID: "34a4ce20-4dfb-4635-8e91-c016747738e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:07:30.658638 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.658614 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4" (OuterVolumeSpecName: "kube-api-access-v8gz4") pod "34a4ce20-4dfb-4635-8e91-c016747738e4" (UID: "34a4ce20-4dfb-4635-8e91-c016747738e4"). InnerVolumeSpecName "kube-api-access-v8gz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:07:30.658744 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.658722 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "272dcb26-94bc-488f-b1d4-f42ce7fcab28" (UID: "272dcb26-94bc-488f-b1d4-f42ce7fcab28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:07:30.658792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.658762 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8" (OuterVolumeSpecName: "kube-api-access-5rvp8") pod "272dcb26-94bc-488f-b1d4-f42ce7fcab28" (UID: "272dcb26-94bc-488f-b1d4-f42ce7fcab28"). InnerVolumeSpecName "kube-api-access-5rvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:07:30.739360 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.739262 2581 generic.go:356] "Generic (PLEG): container finished" podID="34a4ce20-4dfb-4635-8e91-c016747738e4" containerID="be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36" exitCode=0 Jul 24 14:07:30.739360 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.739319 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" Jul 24 14:07:30.739360 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.739347 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" event={"ID":"34a4ce20-4dfb-4635-8e91-c016747738e4","Type":"ContainerDied","Data":"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36"} Jul 24 14:07:30.739661 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.739411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd" event={"ID":"34a4ce20-4dfb-4635-8e91-c016747738e4","Type":"ContainerDied","Data":"6adef82f1ba5e8093d5d1dae7dea85e89c18a39067f669c2f4f70b7818aceb68"} Jul 24 14:07:30.739661 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.739452 2581 scope.go:117] "RemoveContainer" containerID="be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36" Jul 24 14:07:30.740753 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.740731 2581 generic.go:356] "Generic (PLEG): container finished" podID="21424deb-2cfe-47e9-a441-72c21d7f89eb" containerID="fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c" exitCode=0 Jul 24 14:07:30.740850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.740774 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" Jul 24 14:07:30.740850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.740777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" event={"ID":"21424deb-2cfe-47e9-a441-72c21d7f89eb","Type":"ContainerDied","Data":"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c"} Jul 24 14:07:30.740850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.740825 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps" event={"ID":"21424deb-2cfe-47e9-a441-72c21d7f89eb","Type":"ContainerDied","Data":"c0feaefa48823c72133ff92e86a5fb9a53566350c96dd40672787fb20944f1e2"} Jul 24 14:07:30.742319 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.742283 2581 generic.go:356] "Generic (PLEG): container finished" podID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" containerID="e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f" exitCode=0 Jul 24 14:07:30.742319 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.742317 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" event={"ID":"272dcb26-94bc-488f-b1d4-f42ce7fcab28","Type":"ContainerDied","Data":"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f"} Jul 24 14:07:30.742479 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.742330 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" Jul 24 14:07:30.742479 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.742350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf" event={"ID":"272dcb26-94bc-488f-b1d4-f42ce7fcab28","Type":"ContainerDied","Data":"b69963c518a81758955a16bf8deb4171a501bceca11f695c3239b7677993b8f1"} Jul 24 14:07:30.749083 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.749034 2581 scope.go:117] "RemoveContainer" containerID="be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36" Jul 24 14:07:30.749459 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:30.749417 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36\": container with ID starting with be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36 not found: ID does not exist" containerID="be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36" Jul 24 14:07:30.749459 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.749449 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36"} err="failed to get container status \"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36\": rpc error: code = NotFound desc = could not find container \"be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36\": container with ID starting with be23c0d7761f754f924edf76791581271303c533ce992e682b0d24c81e21cb36 not found: ID does not exist" Jul 24 14:07:30.749608 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.749472 2581 scope.go:117] "RemoveContainer" containerID="fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c" Jul 24 14:07:30.757953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.757844 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc7qt\" (UniqueName: \"kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt\") pod \"21424deb-2cfe-47e9-a441-72c21d7f89eb\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " Jul 24 14:07:30.757953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.757888 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume\") pod \"21424deb-2cfe-47e9-a441-72c21d7f89eb\" (UID: \"21424deb-2cfe-47e9-a441-72c21d7f89eb\") " Jul 24 14:07:30.758062 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.757984 2581 scope.go:117] "RemoveContainer" containerID="fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c" Jul 24 14:07:30.758220 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758202 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvp8\" (UniqueName: \"kubernetes.io/projected/272dcb26-94bc-488f-b1d4-f42ce7fcab28-kube-api-access-5rvp8\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.758288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758227 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/272dcb26-94bc-488f-b1d4-f42ce7fcab28-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.758288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758241 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gz4\" (UniqueName: \"kubernetes.io/projected/34a4ce20-4dfb-4635-8e91-c016747738e4-kube-api-access-v8gz4\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.758288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758255 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4ce20-4dfb-4635-8e91-c016747738e4-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.758288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758269 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/272dcb26-94bc-488f-b1d4-f42ce7fcab28-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.758450 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758339 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21424deb-2cfe-47e9-a441-72c21d7f89eb" (UID: "21424deb-2cfe-47e9-a441-72c21d7f89eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:07:30.758450 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:30.758351 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c\": container with ID starting with fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c not found: ID does not exist" containerID="fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c" Jul 24 14:07:30.758450 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758390 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c"} err="failed to get container status \"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c\": rpc error: code = NotFound desc = could not find container \"fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c\": container with ID starting with fc7c5a13f033631fade417df2abdbb90d93a842741f86aead07a9c7f7476b36c not found: ID does not exist" Jul 24 14:07:30.758450 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.758409 2581 scope.go:117] "RemoveContainer" containerID="e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f" Jul 24 14:07:30.760368 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.760345 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt" (OuterVolumeSpecName: "kube-api-access-hc7qt") pod "21424deb-2cfe-47e9-a441-72c21d7f89eb" (UID: "21424deb-2cfe-47e9-a441-72c21d7f89eb"). InnerVolumeSpecName "kube-api-access-hc7qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:07:30.764935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.764900 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:07:30.768360 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.768335 2581 scope.go:117] "RemoveContainer" containerID="e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f" Jul 24 14:07:30.768727 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:30.768704 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f\": container with ID starting with e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f not found: ID does not exist" containerID="e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f" Jul 24 14:07:30.768872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.768732 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f"} err="failed to get container status \"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f\": rpc error: code = NotFound desc = could not find container \"e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f\": container with ID starting with e39b22584c3a533473c2f361569e800889d286f5b70df04c5c1de1b9dab6330f not found: ID does not exist" Jul 24 14:07:30.770185 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.770164 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-35e2fd78-7c98f8b48-l68bd"] Jul 24 14:07:30.784236 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.784201 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:07:30.787827 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.787795 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-35e2fd78-c6fbbb866-n45kf"] Jul 24 14:07:30.859333 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.859291 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7qt\" (UniqueName: \"kubernetes.io/projected/21424deb-2cfe-47e9-a441-72c21d7f89eb-kube-api-access-hc7qt\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.859333 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.859323 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21424deb-2cfe-47e9-a441-72c21d7f89eb-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:30.916436 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.916391 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" path="/var/lib/kubelet/pods/272dcb26-94bc-488f-b1d4-f42ce7fcab28/volumes" Jul 24 14:07:30.916891 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:30.916869 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a4ce20-4dfb-4635-8e91-c016747738e4" path="/var/lib/kubelet/pods/34a4ce20-4dfb-4635-8e91-c016747738e4/volumes" Jul 24 14:07:31.055848 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:31.055760 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:07:31.060077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:31.060048 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-35e2fd78-d8f4b4d68-46xps"] Jul 24 14:07:32.916055 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:32.916011 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21424deb-2cfe-47e9-a441-72c21d7f89eb" path="/var/lib/kubelet/pods/21424deb-2cfe-47e9-a441-72c21d7f89eb/volumes" Jul 24 14:07:34.664151 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:34.664116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:34.806444 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:34.806408 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:07:34.806705 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:34.806679 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" containerID="cri-o://00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5" gracePeriod=30 Jul 24 14:07:34.833786 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:34.833751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.082631 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.082570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.326054 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.326016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.425919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.425886 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.553658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.553625 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.863117 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.863019 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:07:35.864337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864303 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a4ce20-4dfb-4635-8e91-c016747738e4" containerName="payload-processing" Jul 24 14:07:35.864337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864340 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a4ce20-4dfb-4635-8e91-c016747738e4" containerName="payload-processing" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864362 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" containerName="maas-api" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864372 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" containerName="maas-api" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864386 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21424deb-2cfe-47e9-a441-72c21d7f89eb" containerName="payload-pre-processing" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864399 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21424deb-2cfe-47e9-a441-72c21d7f89eb" containerName="payload-pre-processing" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864500 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="272dcb26-94bc-488f-b1d4-f42ce7fcab28" containerName="maas-api" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864513 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21424deb-2cfe-47e9-a441-72c21d7f89eb" containerName="payload-pre-processing" Jul 24 14:07:35.864561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.864522 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a4ce20-4dfb-4635-8e91-c016747738e4" containerName="payload-processing" Jul 24 14:07:35.869427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.869392 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:35.871760 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.871737 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-83602dfb-gw-openshift-default-dockercfg-ftvh9\"" Jul 24 14:07:35.878679 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.878636 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:07:35.893759 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.893727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:35.999903 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:35.999863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:36.002380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956hb\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002596 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002596 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002530 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002596 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002720 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.002720 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.002650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.103894 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.103849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.103917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.103968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.103995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956hb\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104338 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104566 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104566 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.104735 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.104711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.105039 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.105017 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.106548 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.106514 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.106854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.106831 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.116356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.116293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-956hb\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.116481 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.116388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token\") pod \"e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.182031 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.181994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:36.333135 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.333108 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:07:36.335254 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:07:36.335223 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18f3a59_37dc_48b1_b44c_90638c7403db.slice/crio-200a0dce1076889ccfe52b6072cb45892ff33c5ab8529b8133c929742e20fd49 WatchSource:0}: Error finding container 200a0dce1076889ccfe52b6072cb45892ff33c5ab8529b8133c929742e20fd49: Status 404 returned error can't find the container with id 200a0dce1076889ccfe52b6072cb45892ff33c5ab8529b8133c929742e20fd49 Jul 24 14:07:36.337320 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.337287 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:36.337392 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.337361 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:36.337429 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.337395 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:36.778778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.778742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerStarted","Data":"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb"} Jul 24 14:07:36.778983 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.778802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerStarted","Data":"200a0dce1076889ccfe52b6072cb45892ff33c5ab8529b8133c929742e20fd49"} Jul 24 14:07:36.798695 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:36.798639 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podStartSLOduration=1.798620136 podStartE2EDuration="1.798620136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:36.797393594 +0000 UTC m=+1750.393957648" watchObservedRunningTime="2026-07-24 14:07:36.798620136 +0000 UTC m=+1750.395184190" Jul 24 14:07:37.182966 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:37.182925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:37.188470 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:37.188447 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:37.784216 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:37.784187 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:37.785230 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:37.785204 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:38.310106 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.310061 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:38.369946 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.369845 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:07:38.374178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.374149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.376420 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.376393 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:07:38.376692 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.376677 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-83602dfb-serving-cert\"" Jul 24 14:07:38.383036 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.382984 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:07:38.412856 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.412825 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:38.427959 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.427923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.428209 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.428004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.428209 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.428052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5h5x\" (UniqueName: \"kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.528633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.528595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.528833 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.528655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5h5x\" (UniqueName: \"kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.528833 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.528683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.528833 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:38.528691 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-serving-cert: secret "maas-api-e2e-migrate-83602dfb-serving-cert" not found Jul 24 14:07:38.528833 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:38.528792 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls podName:59a97a7a-6c9d-45aa-8d47-1304c90e748a nodeName:}" failed. No retries permitted until 2026-07-24 14:07:39.028774326 +0000 UTC m=+1752.625338359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls") pod "maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" (UID: "59a97a7a-6c9d-45aa-8d47-1304c90e748a") : secret "maas-api-e2e-migrate-83602dfb-serving-cert" not found Jul 24 14:07:38.529374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.529356 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.537077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.537050 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5h5x\" (UniqueName: \"kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:38.678870 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:38.678838 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:39.011220 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.011113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:39.032567 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.032533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:39.035522 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.035479 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") pod \"maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:39.112489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.112459 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:07:39.121173 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.121138 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:07:39.124919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.124894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.127147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.127129 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-83602dfb\"" Jul 24 14:07:39.127232 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.127142 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-83602dfb-dockercfg-l5pfl\"" Jul 24 14:07:39.133245 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.133220 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.133383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.133251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5ff\" (UniqueName: \"kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.134379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.134356 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:07:39.138111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.138089 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.138342 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.138315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:07:39.148676 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.148646 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:07:39.235111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.235051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.235111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.235098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljdgq\" (UniqueName: \"kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.235382 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.235198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.235382 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.235231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5ff\" (UniqueName: \"kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.236368 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.236341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.243974 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.243943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5ff\" (UniqueName: \"kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff\") pod \"payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.287882 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.287786 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:39.336133 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.336095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.336644 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.336145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdgq\" (UniqueName: \"kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.336808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.336790 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.344845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.344800 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdgq\" (UniqueName: \"kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq\") pod \"payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.436822 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.436750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.439525 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.439489 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:07:39.441688 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:07:39.441663 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a97a7a_6c9d_45aa_8d47_1304c90e748a.slice/crio-d8a9f684c83cd135da34fb77beda2f4d57e2043b3fefa8b0a010446fa5dce265 WatchSource:0}: Error finding container d8a9f684c83cd135da34fb77beda2f4d57e2043b3fefa8b0a010446fa5dce265: Status 404 returned error can't find the container with id d8a9f684c83cd135da34fb77beda2f4d57e2043b3fefa8b0a010446fa5dce265 Jul 24 14:07:39.450944 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.450919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:39.597170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.596814 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:07:39.600641 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:07:39.600605 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf799c05b_eb19_4494_888b_c48174d87151.slice/crio-a8b38e4cfaa2f977aa17ae85ea3eebf96bfcb2d1d9e903febb1a2a32b32899df WatchSource:0}: Error finding container a8b38e4cfaa2f977aa17ae85ea3eebf96bfcb2d1d9e903febb1a2a32b32899df: Status 404 returned error can't find the container with id a8b38e4cfaa2f977aa17ae85ea3eebf96bfcb2d1d9e903febb1a2a32b32899df Jul 24 14:07:39.629793 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.629688 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:07:39.632833 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:07:39.632796 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac0d36a_188b_46ab_8f29_19c41f66720d.slice/crio-3b2e6625b39415d55ee0b600af9357f5bcff83899adc2afaa1a4dc705ec20080 WatchSource:0}: Error finding container 3b2e6625b39415d55ee0b600af9357f5bcff83899adc2afaa1a4dc705ec20080: Status 404 returned error can't find the container with id 3b2e6625b39415d55ee0b600af9357f5bcff83899adc2afaa1a4dc705ec20080 Jul 24 14:07:39.803232 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.803193 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" event={"ID":"f799c05b-eb19-4494-888b-c48174d87151","Type":"ContainerStarted","Data":"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936"} Jul 24 14:07:39.803375 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.803295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" event={"ID":"f799c05b-eb19-4494-888b-c48174d87151","Type":"ContainerStarted","Data":"a8b38e4cfaa2f977aa17ae85ea3eebf96bfcb2d1d9e903febb1a2a32b32899df"} Jul 24 14:07:39.805045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.805018 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" event={"ID":"8ac0d36a-188b-46ab-8f29-19c41f66720d","Type":"ContainerStarted","Data":"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49"} Jul 24 14:07:39.805178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.805065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" event={"ID":"8ac0d36a-188b-46ab-8f29-19c41f66720d","Type":"ContainerStarted","Data":"3b2e6625b39415d55ee0b600af9357f5bcff83899adc2afaa1a4dc705ec20080"} Jul 24 14:07:39.805688 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.805450 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:39.806331 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.806305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" event={"ID":"59a97a7a-6c9d-45aa-8d47-1304c90e748a","Type":"ContainerStarted","Data":"d8a9f684c83cd135da34fb77beda2f4d57e2043b3fefa8b0a010446fa5dce265"} Jul 24 14:07:39.822166 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.822107 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" podStartSLOduration=0.822089844 podStartE2EDuration="822.089844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:39.821125172 +0000 UTC m=+1753.417689227" watchObservedRunningTime="2026-07-24 14:07:39.822089844 +0000 UTC m=+1753.418653893" Jul 24 14:07:39.836864 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:39.836804 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" podStartSLOduration=0.836783484 podStartE2EDuration="836.783484ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:39.836602528 +0000 UTC m=+1753.433166586" watchObservedRunningTime="2026-07-24 14:07:39.836783484 +0000 UTC m=+1753.433347542" Jul 24 14:07:40.059912 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.059878 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:40.144261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144164 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144211 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144234 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144289 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144366 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144388 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144417 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqblg\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert\") pod \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\" (UID: \"47b3998b-ab3d-41d2-a0a5-2daf9135e25f\") " Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144458 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:07:40.144545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144519 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:07:40.144987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144636 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:07:40.144987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144843 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.144987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144863 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.144987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144872 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:07:40.144987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.144879 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.145347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.145302 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data" (OuterVolumeSpecName: "istio-data") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:07:40.146786 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.146763 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg" (OuterVolumeSpecName: "kube-api-access-wqblg") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "kube-api-access-wqblg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:07:40.146884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.146790 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:07:40.147077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.147062 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token" (OuterVolumeSpecName: "istio-token") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:07:40.147118 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.147067 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47b3998b-ab3d-41d2-a0a5-2daf9135e25f" (UID: "47b3998b-ab3d-41d2-a0a5-2daf9135e25f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:07:40.246095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246061 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.246095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246090 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.246095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246102 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.246347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246113 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.246347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246127 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqblg\" (UniqueName: \"kubernetes.io/projected/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-kube-api-access-wqblg\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.246347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.246140 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3998b-ab3d-41d2-a0a5-2daf9135e25f-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:07:40.815276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.815173 2581 generic.go:356] "Generic (PLEG): container finished" podID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerID="00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5" exitCode=0 Jul 24 14:07:40.815276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.815218 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" Jul 24 14:07:40.815276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.815260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerDied","Data":"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5"} Jul 24 14:07:40.815872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.815315 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk" event={"ID":"47b3998b-ab3d-41d2-a0a5-2daf9135e25f","Type":"ContainerDied","Data":"c0733dde075e4562314553ac205f1cb4121c767465ce2cc5ec2c8b957b5a46d9"} Jul 24 14:07:40.815872 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.815346 2581 scope.go:117] "RemoveContainer" containerID="00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5" Jul 24 14:07:40.817155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.817122 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" event={"ID":"59a97a7a-6c9d-45aa-8d47-1304c90e748a","Type":"ContainerStarted","Data":"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183"} Jul 24 14:07:40.817279 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.817245 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:40.825838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.825814 2581 scope.go:117] "RemoveContainer" containerID="791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9" Jul 24 14:07:40.834489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.834470 2581 scope.go:117] "RemoveContainer" containerID="00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5" Jul 24 14:07:40.834893 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:40.834839 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5\": container with ID starting with 00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5 not found: ID does not exist" containerID="00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5" Jul 24 14:07:40.835071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.834887 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5"} err="failed to get container status \"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5\": rpc error: code = NotFound desc = could not find container \"00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5\": container with ID starting with 00ec251182355c9ed9658ed9078f78cd87771f8298fa72c1048264d828303fc5 not found: ID does not exist" Jul 24 14:07:40.835071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.834912 2581 scope.go:117] "RemoveContainer" containerID="791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9" Jul 24 14:07:40.835350 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:07:40.835303 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9\": container with ID starting with 791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9 not found: ID does not exist" containerID="791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9" Jul 24 14:07:40.835460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.835344 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9"} err="failed to get container status \"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9\": rpc error: code = NotFound desc = could not find container \"791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9\": container with ID starting with 791e5e3a2b02938bc22ba9d357baf4935f1f82c959547150cabb7c586de7fca9 not found: ID does not exist" Jul 24 14:07:40.837175 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.837134 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" podStartSLOduration=2.4809233969999998 podStartE2EDuration="2.837120539s" totalImagesPullingTime="356.197142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:40.835065902 +0000 UTC m=+1754.431629954" watchObservedRunningTime="2026-07-24 14:07:40.837120539 +0000 UTC m=+1754.433684594" Jul 24 14:07:40.849937 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.849903 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:07:40.853689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.853663 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-35e2fd78-openshift-default-5c67854b64-rzvpk"] Jul 24 14:07:40.917446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:40.917396 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" path="/var/lib/kubelet/pods/47b3998b-ab3d-41d2-a0a5-2daf9135e25f/volumes" Jul 24 14:07:41.823049 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:41.823018 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:47.833611 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:47.833561 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:07:50.820004 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:50.819968 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:07:51.825581 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:51.825546 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:07:53.825809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:53.825764 2581 patch_prober.go:28] interesting pod/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:07:53.826233 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:53.825849 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:07:54.899786 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.899755 2581 generic.go:356] "Generic (PLEG): container finished" podID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerID="3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb" exitCode=0 Jul 24 14:07:54.900327 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.899826 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerDied","Data":"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb"} Jul 24 14:07:54.900404 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.900324 2581 scope.go:117] "RemoveContainer" containerID="3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb" Jul 24 14:07:54.901343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.901311 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:54.901405 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.901381 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:54.901445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:54.901409 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:07:55.907403 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:55.907368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerStarted","Data":"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0"} Jul 24 14:07:56.182825 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:56.182788 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:07:56.183286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:56.183195 2581 patch_prober.go:28] interesting pod/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 14:07:56.183479 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:56.183454 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 14:07:57.183140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:57.183080 2581 patch_prober.go:28] interesting pod/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 14:07:57.183849 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:57.183789 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 14:07:58.183019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:58.182977 2581 patch_prober.go:28] interesting pod/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 14:07:58.183353 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:58.183324 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 14:07:59.183182 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:59.183141 2581 patch_prober.go:28] interesting pod/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 14:07:59.183529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:07:59.183485 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 14:08:00.186148 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:00.186115 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:08:00.186655 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:00.186397 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:08:00.187203 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:00.187183 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:08:02.518635 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:02.518597 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:13.960886 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:13.960853 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:08:13.961297 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:13.961069 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" podUID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" containerName="maas-api" containerID="cri-o://d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183" gracePeriod=30 Jul 24 14:08:13.987662 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:13.987623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:14.027686 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.027655 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:08:14.027946 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.027918 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" podUID="8ac0d36a-188b-46ab-8f29-19c41f66720d" containerName="payload-processing" containerID="cri-o://725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49" gracePeriod=30 Jul 24 14:08:14.036314 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.036238 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:08:14.036717 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.036484 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" podUID="f799c05b-eb19-4494-888b-c48174d87151" containerName="payload-pre-processing" containerID="cri-o://4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936" gracePeriod=30 Jul 24 14:08:14.074943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.074901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:14.148076 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.148034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:14.335359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.335335 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:08:14.338833 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.338804 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:08:14.367447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.367416 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:08:14.462331 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462295 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume\") pod \"8ac0d36a-188b-46ab-8f29-19c41f66720d\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " Jul 24 14:08:14.462540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462362 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume\") pod \"f799c05b-eb19-4494-888b-c48174d87151\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " Jul 24 14:08:14.462540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462389 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca\") pod \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " Jul 24 14:08:14.462540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462409 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5h5x\" (UniqueName: \"kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x\") pod \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " Jul 24 14:08:14.462540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462439 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") pod \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\" (UID: \"59a97a7a-6c9d-45aa-8d47-1304c90e748a\") " Jul 24 14:08:14.462540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462462 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljdgq\" (UniqueName: \"kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq\") pod \"8ac0d36a-188b-46ab-8f29-19c41f66720d\" (UID: \"8ac0d36a-188b-46ab-8f29-19c41f66720d\") " Jul 24 14:08:14.462860 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462628 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp5ff\" (UniqueName: \"kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff\") pod \"f799c05b-eb19-4494-888b-c48174d87151\" (UID: \"f799c05b-eb19-4494-888b-c48174d87151\") " Jul 24 14:08:14.462860 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462835 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ac0d36a-188b-46ab-8f29-19c41f66720d" (UID: "8ac0d36a-188b-46ab-8f29-19c41f66720d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:14.462978 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.462841 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f799c05b-eb19-4494-888b-c48174d87151" (UID: "f799c05b-eb19-4494-888b-c48174d87151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:14.465192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.463561 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59a97a7a-6c9d-45aa-8d47-1304c90e748a" (UID: "59a97a7a-6c9d-45aa-8d47-1304c90e748a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:14.465192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.465036 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq" (OuterVolumeSpecName: "kube-api-access-ljdgq") pod "8ac0d36a-188b-46ab-8f29-19c41f66720d" (UID: "8ac0d36a-188b-46ab-8f29-19c41f66720d"). InnerVolumeSpecName "kube-api-access-ljdgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:14.465410 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.465317 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x" (OuterVolumeSpecName: "kube-api-access-s5h5x") pod "59a97a7a-6c9d-45aa-8d47-1304c90e748a" (UID: "59a97a7a-6c9d-45aa-8d47-1304c90e748a"). InnerVolumeSpecName "kube-api-access-s5h5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:14.465541 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.465508 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59a97a7a-6c9d-45aa-8d47-1304c90e748a" (UID: "59a97a7a-6c9d-45aa-8d47-1304c90e748a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:08:14.465665 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.465552 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff" (OuterVolumeSpecName: "kube-api-access-xp5ff") pod "f799c05b-eb19-4494-888b-c48174d87151" (UID: "f799c05b-eb19-4494-888b-c48174d87151"). InnerVolumeSpecName "kube-api-access-xp5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563364 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f799c05b-eb19-4494-888b-c48174d87151-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563397 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a97a7a-6c9d-45aa-8d47-1304c90e748a-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563409 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5h5x\" (UniqueName: \"kubernetes.io/projected/59a97a7a-6c9d-45aa-8d47-1304c90e748a-kube-api-access-s5h5x\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563423 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a97a7a-6c9d-45aa-8d47-1304c90e748a-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563436 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljdgq\" (UniqueName: \"kubernetes.io/projected/8ac0d36a-188b-46ab-8f29-19c41f66720d-kube-api-access-ljdgq\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563448 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5ff\" (UniqueName: \"kubernetes.io/projected/f799c05b-eb19-4494-888b-c48174d87151-kube-api-access-xp5ff\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:14.563460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:14.563459 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac0d36a-188b-46ab-8f29-19c41f66720d-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:15.004028 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.003991 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ac0d36a-188b-46ab-8f29-19c41f66720d" containerID="725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49" exitCode=0 Jul 24 14:08:15.004517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.004053 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" Jul 24 14:08:15.004517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.004083 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" event={"ID":"8ac0d36a-188b-46ab-8f29-19c41f66720d","Type":"ContainerDied","Data":"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49"} Jul 24 14:08:15.004517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.004141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw" event={"ID":"8ac0d36a-188b-46ab-8f29-19c41f66720d","Type":"ContainerDied","Data":"3b2e6625b39415d55ee0b600af9357f5bcff83899adc2afaa1a4dc705ec20080"} Jul 24 14:08:15.004517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.004175 2581 scope.go:117] "RemoveContainer" containerID="725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49" Jul 24 14:08:15.005825 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.005797 2581 generic.go:356] "Generic (PLEG): container finished" podID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" containerID="d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183" exitCode=0 Jul 24 14:08:15.005943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.005854 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" Jul 24 14:08:15.005943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.005920 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" event={"ID":"59a97a7a-6c9d-45aa-8d47-1304c90e748a","Type":"ContainerDied","Data":"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183"} Jul 24 14:08:15.006055 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.005983 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85" event={"ID":"59a97a7a-6c9d-45aa-8d47-1304c90e748a","Type":"ContainerDied","Data":"d8a9f684c83cd135da34fb77beda2f4d57e2043b3fefa8b0a010446fa5dce265"} Jul 24 14:08:15.007604 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.007556 2581 generic.go:356] "Generic (PLEG): container finished" podID="f799c05b-eb19-4494-888b-c48174d87151" containerID="4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936" exitCode=0 Jul 24 14:08:15.007722 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.007617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" event={"ID":"f799c05b-eb19-4494-888b-c48174d87151","Type":"ContainerDied","Data":"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936"} Jul 24 14:08:15.007722 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.007674 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" event={"ID":"f799c05b-eb19-4494-888b-c48174d87151","Type":"ContainerDied","Data":"a8b38e4cfaa2f977aa17ae85ea3eebf96bfcb2d1d9e903febb1a2a32b32899df"} Jul 24 14:08:15.007722 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.007703 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg" Jul 24 14:08:15.014529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.014507 2581 scope.go:117] "RemoveContainer" containerID="725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49" Jul 24 14:08:15.014887 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:15.014865 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49\": container with ID starting with 725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49 not found: ID does not exist" containerID="725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49" Jul 24 14:08:15.014968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.014897 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49"} err="failed to get container status \"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49\": rpc error: code = NotFound desc = could not find container \"725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49\": container with ID starting with 725261d687e6fa8a2ff03eb9750cfcc26d3466b7ed2721747375957470d3de49 not found: ID does not exist" Jul 24 14:08:15.014968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.014915 2581 scope.go:117] "RemoveContainer" containerID="d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183" Jul 24 14:08:15.023943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.023920 2581 scope.go:117] "RemoveContainer" containerID="d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183" Jul 24 14:08:15.024306 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:15.024287 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183\": container with ID starting with d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183 not found: ID does not exist" containerID="d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183" Jul 24 14:08:15.024359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.024314 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183"} err="failed to get container status \"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183\": rpc error: code = NotFound desc = could not find container \"d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183\": container with ID starting with d38efeb7254900734edb54513693e377347dc956d51df3909abcd1408afcf183 not found: ID does not exist" Jul 24 14:08:15.024359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.024333 2581 scope.go:117] "RemoveContainer" containerID="4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936" Jul 24 14:08:15.030808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.030777 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:08:15.033663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.033637 2581 scope.go:117] "RemoveContainer" containerID="4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936" Jul 24 14:08:15.033984 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:15.033965 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936\": container with ID starting with 4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936 not found: ID does not exist" containerID="4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936" Jul 24 14:08:15.034052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.033991 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936"} err="failed to get container status \"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936\": rpc error: code = NotFound desc = could not find container \"4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936\": container with ID starting with 4776e1a9b979fb4c3faec3cb828facc667c5fdff966335d658ddd5569e28e936 not found: ID does not exist" Jul 24 14:08:15.035474 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.035448 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-83602dfb-79c567888f-5sclw"] Jul 24 14:08:15.048380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.048335 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:08:15.052904 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.052871 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-83602dfb-b4c87d6f6-7x8hg"] Jul 24 14:08:15.064171 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.064137 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:08:15.068636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:15.068609 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-83602dfb-5c4dc99474-9qx85"] Jul 24 14:08:16.917477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:16.917432 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" path="/var/lib/kubelet/pods/59a97a7a-6c9d-45aa-8d47-1304c90e748a/volumes" Jul 24 14:08:16.917998 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:16.917978 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac0d36a-188b-46ab-8f29-19c41f66720d" path="/var/lib/kubelet/pods/8ac0d36a-188b-46ab-8f29-19c41f66720d/volumes" Jul 24 14:08:16.918353 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:16.918334 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f799c05b-eb19-4494-888b-c48174d87151" path="/var/lib/kubelet/pods/f799c05b-eb19-4494-888b-c48174d87151/volumes" Jul 24 14:08:18.433551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:18.433470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:18.579404 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:18.579366 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:08:18.579762 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:18.579729 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" containerID="cri-o://1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0" gracePeriod=30 Jul 24 14:08:18.603317 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:18.603286 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:18.854029 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:18.853946 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:19.129354 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.129276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:19.418001 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.417971 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:08:19.418398 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418381 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418400 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418415 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" containerName="maas-api" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418423 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" containerName="maas-api" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418442 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac0d36a-188b-46ab-8f29-19c41f66720d" containerName="payload-processing" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418450 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac0d36a-188b-46ab-8f29-19c41f66720d" containerName="payload-processing" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418474 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f799c05b-eb19-4494-888b-c48174d87151" containerName="payload-pre-processing" Jul 24 14:08:19.418502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418482 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f799c05b-eb19-4494-888b-c48174d87151" containerName="payload-pre-processing" Jul 24 14:08:19.418874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418645 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:19.418874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418660 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac0d36a-188b-46ab-8f29-19c41f66720d" containerName="payload-processing" Jul 24 14:08:19.418874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418671 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f799c05b-eb19-4494-888b-c48174d87151" containerName="payload-pre-processing" Jul 24 14:08:19.418874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418679 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a97a7a-6c9d-45aa-8d47-1304c90e748a" containerName="maas-api" Jul 24 14:08:19.418874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.418687 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:19.423449 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.423422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.425881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.425861 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-388084cb-openshift-default-dockercfg-djkcn\"" Jul 24 14:08:19.434869 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.434837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:08:19.442954 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.442925 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:19.505836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.505796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506006 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.505850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506006 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.505939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506006 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.505967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506006 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.505997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.506039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.506124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpg6\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.506184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.506248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.506230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607716 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607677 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607716 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpg6\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.607995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.607953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608363 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608295 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608419 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608382 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608562 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608540 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608810 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608794 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.608915 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.608895 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.610172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.610152 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.610407 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.610391 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.617110 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.617079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpg6\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.617248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.617115 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token\") pod \"e2e-ait-388084cb-openshift-default-574f98d-wmkst\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.735815 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.735713 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:19.904986 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.904959 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:08:19.907139 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:08:19.907111 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c7af01_7a55_46c2_9e95_ba18c792d7dc.slice/crio-dbce3191db58f05973e95111dcea366c8acc7e36e1edee32379fe822fa7a7820 WatchSource:0}: Error finding container dbce3191db58f05973e95111dcea366c8acc7e36e1edee32379fe822fa7a7820: Status 404 returned error can't find the container with id dbce3191db58f05973e95111dcea366c8acc7e36e1edee32379fe822fa7a7820 Jul 24 14:08:19.909861 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.909822 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:19.909952 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.909915 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:19.909992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:19.909956 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:20.036781 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.036678 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerStarted","Data":"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e"} Jul 24 14:08:20.036781 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.036746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerStarted","Data":"dbce3191db58f05973e95111dcea366c8acc7e36e1edee32379fe822fa7a7820"} Jul 24 14:08:20.083147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.083089 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podStartSLOduration=1.083072357 podStartE2EDuration="1.083072357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:08:20.082281452 +0000 UTC m=+1793.678845507" watchObservedRunningTime="2026-07-24 14:08:20.083072357 +0000 UTC m=+1793.679636411" Jul 24 14:08:20.087417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.087383 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:20.736765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.736723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:20.742078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:20.742049 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:21.044423 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.044322 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:21.045399 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.045382 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:21.435667 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.435634 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:21.551651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.551609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:21.552130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.552105 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:21.552130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.552128 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b3998b-ab3d-41d2-a0a5-2daf9135e25f" containerName="istio-proxy" Jul 24 14:08:21.552773 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.552752 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:21.555345 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.555325 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.557751 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.557728 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-388084cb-serving-cert\"" Jul 24 14:08:21.558059 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.558045 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:08:21.569785 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.569747 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:21.672418 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.672388 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:21.728663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.728559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qjx\" (UniqueName: \"kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.728822 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.728673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.728822 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.728713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.791389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.791350 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:21.829403 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.829370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.829403 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.829407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.829644 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.829481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qjx\" (UniqueName: \"kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.830217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.830197 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.831889 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.831868 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.838399 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.838370 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qjx\" (UniqueName: \"kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx\") pod \"maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:21.871375 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:21.871333 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:22.014328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.014201 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:22.016532 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:08:22.016499 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d648725_a236_44ca_9463_eb7d4b06fa3b.slice/crio-a6e58765cca79ac79ab980f825b4ee990accf9ffa877b34c637bb3f80dc36b3c WatchSource:0}: Error finding container a6e58765cca79ac79ab980f825b4ee990accf9ffa877b34c637bb3f80dc36b3c: Status 404 returned error can't find the container with id a6e58765cca79ac79ab980f825b4ee990accf9ffa877b34c637bb3f80dc36b3c Jul 24 14:08:22.049852 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.049814 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" event={"ID":"9d648725-a236-44ca-9463-eb7d4b06fa3b","Type":"ContainerStarted","Data":"a6e58765cca79ac79ab980f825b4ee990accf9ffa877b34c637bb3f80dc36b3c"} Jul 24 14:08:22.189798 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.189762 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:22.287076 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.286991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:22.316693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.316659 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:22.323509 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.323457 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.326121 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.326094 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-388084cb\"" Jul 24 14:08:22.326267 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.326094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-388084cb-dockercfg-qhcnc\"" Jul 24 14:08:22.334159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.334132 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:22.435546 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.435508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.435546 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.435547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56nbs\" (UniqueName: \"kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.537150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.537055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.537150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.537093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56nbs\" (UniqueName: \"kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.537889 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.537869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.549012 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.548979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56nbs\" (UniqueName: \"kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs\") pod \"payload-processing-e2e-ait-388084cb-7b5c586d68-988b4\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.635515 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.635469 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:22.997706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:22.997677 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:23.000020 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:08:22.999987 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1af8d7_359e_4bc0_9d25_655823ce8a3c.slice/crio-7f93efe503e0980f2685b23b87f4bacb6e088475234d915de156b1e93abbffc1 WatchSource:0}: Error finding container 7f93efe503e0980f2685b23b87f4bacb6e088475234d915de156b1e93abbffc1: Status 404 returned error can't find the container with id 7f93efe503e0980f2685b23b87f4bacb6e088475234d915de156b1e93abbffc1 Jul 24 14:08:23.057300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.057269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" event={"ID":"8e1af8d7-359e-4bc0-9d25-655823ce8a3c","Type":"ContainerStarted","Data":"7f93efe503e0980f2685b23b87f4bacb6e088475234d915de156b1e93abbffc1"} Jul 24 14:08:23.059269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.059244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" event={"ID":"9d648725-a236-44ca-9463-eb7d4b06fa3b","Type":"ContainerStarted","Data":"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f"} Jul 24 14:08:23.059470 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.059450 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:23.078848 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.078790 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" podStartSLOduration=1.639909072 podStartE2EDuration="2.07877013s" totalImagesPullingTime="438.861058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:08:23.076666705 +0000 UTC m=+1796.673230786" watchObservedRunningTime="2026-07-24 14:08:23.07877013 +0000 UTC m=+1796.675334184" Jul 24 14:08:23.850774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.850690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:08:23.951819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.951770 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-956hb\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952032 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.951871 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952109 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952033 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952109 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952097 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952128 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952160 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952194 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952369 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952226 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952369 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952257 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy\") pod \"a18f3a59-37dc-48b1-b44c-90638c7403db\" (UID: \"a18f3a59-37dc-48b1-b44c-90638c7403db\") " Jul 24 14:08:23.952369 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952265 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:08:23.952517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952400 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:08:23.952517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:08:23.952669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952623 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:23.952669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952636 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:23.952669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952658 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:23.952823 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.952673 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:23.953031 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.953009 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data" (OuterVolumeSpecName: "istio-data") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:08:23.954288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.954254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:08:23.954288 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.954278 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb" (OuterVolumeSpecName: "kube-api-access-956hb") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "kube-api-access-956hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:23.954574 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.954555 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token" (OuterVolumeSpecName: "istio-token") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:23.954650 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:23.954577 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a18f3a59-37dc-48b1-b44c-90638c7403db" (UID: "a18f3a59-37dc-48b1-b44c-90638c7403db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:08:24.053300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053263 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.053300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053292 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18f3a59-37dc-48b1-b44c-90638c7403db-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.053300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053302 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.053881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053319 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956hb\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-kube-api-access-956hb\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.053881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053332 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.053881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.053340 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18f3a59-37dc-48b1-b44c-90638c7403db-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:24.066957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.066917 2581 generic.go:356] "Generic (PLEG): container finished" podID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerID="1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0" exitCode=0 Jul 24 14:08:24.067137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.066987 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" Jul 24 14:08:24.067137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.066983 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerDied","Data":"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0"} Jul 24 14:08:24.067242 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.067141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt" event={"ID":"a18f3a59-37dc-48b1-b44c-90638c7403db","Type":"ContainerDied","Data":"200a0dce1076889ccfe52b6072cb45892ff33c5ab8529b8133c929742e20fd49"} Jul 24 14:08:24.067242 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.067177 2581 scope.go:117] "RemoveContainer" containerID="1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0" Jul 24 14:08:24.069550 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.069465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" event={"ID":"8e1af8d7-359e-4bc0-9d25-655823ce8a3c","Type":"ContainerStarted","Data":"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73"} Jul 24 14:08:24.070162 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.070140 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:24.079828 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.079808 2581 scope.go:117] "RemoveContainer" containerID="3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb" Jul 24 14:08:24.089940 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.089919 2581 scope.go:117] "RemoveContainer" containerID="1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0" Jul 24 14:08:24.090358 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:24.090331 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0\": container with ID starting with 1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0 not found: ID does not exist" containerID="1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0" Jul 24 14:08:24.090494 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.090369 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0"} err="failed to get container status \"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0\": rpc error: code = NotFound desc = could not find container \"1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0\": container with ID starting with 1d786598bb0c4feac83406c6c6f9ad0e834a227fe9863e342a49e857da78d7a0 not found: ID does not exist" Jul 24 14:08:24.090494 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.090395 2581 scope.go:117] "RemoveContainer" containerID="3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb" Jul 24 14:08:24.090733 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:24.090709 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb\": container with ID starting with 3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb not found: ID does not exist" containerID="3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb" Jul 24 14:08:24.090802 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.090742 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb"} err="failed to get container status \"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb\": rpc error: code = NotFound desc = could not find container \"3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb\": container with ID starting with 3d6db02e18081397c9cfad401087cd171c6fd7d24b5402f6d56dcbc62a4752eb not found: ID does not exist" Jul 24 14:08:24.092100 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.092061 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" podStartSLOduration=2.092047865 podStartE2EDuration="2.092047865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:08:24.089890871 +0000 UTC m=+1797.686454929" watchObservedRunningTime="2026-07-24 14:08:24.092047865 +0000 UTC m=+1797.688611921" Jul 24 14:08:24.104089 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.104011 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:08:24.109937 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.109911 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-83602dfb-gw-openshift-default-6c755445b-8kldt"] Jul 24 14:08:24.916739 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:24.916708 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" path="/var/lib/kubelet/pods/a18f3a59-37dc-48b1-b44c-90638c7403db/volumes" Jul 24 14:08:29.076944 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:29.076907 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:36.084864 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:36.084833 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:36.169711 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:36.169675 2581 patch_prober.go:28] interesting pod/e2e-ait-388084cb-openshift-default-574f98d-wmkst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:33590->10.134.0.58:15021: read: connection reset by peer" start-of-body= Jul 24 14:08:36.169880 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:36.169761 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:33590->10.134.0.58:15021: read: connection reset by peer" Jul 24 14:08:37.144344 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.144309 2581 generic.go:356] "Generic (PLEG): container finished" podID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerID="d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e" exitCode=0 Jul 24 14:08:37.144776 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.144353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerDied","Data":"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e"} Jul 24 14:08:37.144776 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.144766 2581 scope.go:117] "RemoveContainer" containerID="d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e" Jul 24 14:08:37.145812 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.145781 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:37.145917 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.145850 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:37.145917 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:37.145887 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:08:38.151753 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:38.151715 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerStarted","Data":"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788"} Jul 24 14:08:38.736962 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:38.736921 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:38.737166 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:38.737138 2581 patch_prober.go:28] interesting pod/e2e-ait-388084cb-openshift-default-574f98d-wmkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 14:08:38.737221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:38.737194 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 14:08:39.736990 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:39.736952 2581 patch_prober.go:28] interesting pod/e2e-ait-388084cb-openshift-default-574f98d-wmkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 14:08:39.737432 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:39.737019 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 14:08:40.736189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:40.736153 2581 patch_prober.go:28] interesting pod/e2e-ait-388084cb-openshift-default-574f98d-wmkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 14:08:40.736385 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:40.736207 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 14:08:41.737220 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:41.737183 2581 patch_prober.go:28] interesting pod/e2e-ait-388084cb-openshift-default-574f98d-wmkst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 14:08:41.737661 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:41.737241 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 14:08:42.740176 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:42.740146 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:42.740571 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:42.740481 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:42.741374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:42.741355 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:08:47.409374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.409341 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb"] Jul 24 14:08:47.409792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.409743 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:08:47.409792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.409754 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:08:47.409866 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.409824 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:08:47.412643 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.412625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:47.415280 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.415257 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:08:47.422572 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.422542 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb"] Jul 24 14:08:47.446150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.446098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-388084cb-mmjrb\" (UID: \"6ea8fd77-e757-4ede-af8a-632cd0ebce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:47.546665 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.546620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-388084cb-mmjrb\" (UID: \"6ea8fd77-e757-4ede-af8a-632cd0ebce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:47.547277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.547253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-388084cb-mmjrb\" (UID: \"6ea8fd77-e757-4ede-af8a-632cd0ebce58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:47.724351 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.724314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:47.864486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:47.864458 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb"] Jul 24 14:08:47.866961 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:08:47.866932 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea8fd77_e757_4ede_af8a_632cd0ebce58.slice/crio-aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a WatchSource:0}: Error finding container aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a: Status 404 returned error can't find the container with id aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a Jul 24 14:08:48.206027 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:48.205994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" event={"ID":"6ea8fd77-e757-4ede-af8a-632cd0ebce58","Type":"ContainerStarted","Data":"92461aed939abb7b5e3ed83ec01301d3ec21e61207e808ab2b4daa82708f3816"} Jul 24 14:08:48.206221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:48.206051 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" event={"ID":"6ea8fd77-e757-4ede-af8a-632cd0ebce58","Type":"ContainerStarted","Data":"aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a"} Jul 24 14:08:48.220565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:48.220509 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" podStartSLOduration=1.220493191 podStartE2EDuration="1.220493191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:08:48.219865049 +0000 UTC m=+1821.816429103" watchObservedRunningTime="2026-07-24 14:08:48.220493191 +0000 UTC m=+1821.817057246" Jul 24 14:08:49.213322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:49.213286 2581 generic.go:356] "Generic (PLEG): container finished" podID="6ea8fd77-e757-4ede-af8a-632cd0ebce58" containerID="92461aed939abb7b5e3ed83ec01301d3ec21e61207e808ab2b4daa82708f3816" exitCode=0 Jul 24 14:08:49.213752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:49.213377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" event={"ID":"6ea8fd77-e757-4ede-af8a-632cd0ebce58","Type":"ContainerDied","Data":"92461aed939abb7b5e3ed83ec01301d3ec21e61207e808ab2b4daa82708f3816"} Jul 24 14:08:50.357847 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:50.357819 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:50.371391 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:50.371339 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca\") pod \"6ea8fd77-e757-4ede-af8a-632cd0ebce58\" (UID: \"6ea8fd77-e757-4ede-af8a-632cd0ebce58\") " Jul 24 14:08:50.371750 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:50.371725 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ea8fd77-e757-4ede-af8a-632cd0ebce58" (UID: "6ea8fd77-e757-4ede-af8a-632cd0ebce58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:50.472563 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:50.472473 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea8fd77-e757-4ede-af8a-632cd0ebce58-maas-api-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:51.224828 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:51.224735 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" Jul 24 14:08:51.224828 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:51.224767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb" event={"ID":"6ea8fd77-e757-4ede-af8a-632cd0ebce58","Type":"ContainerDied","Data":"aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a"} Jul 24 14:08:51.225026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:51.224830 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea5b5df74d132de077a81452eeae36e452942d8615f13397e61c9ce3ee3120a" Jul 24 14:08:57.857101 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:57.857062 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:57.857943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:57.857886 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" podUID="9d648725-a236-44ca-9463-eb7d4b06fa3b" containerName="maas-api" containerID="cri-o://d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f" gracePeriod=30 Jul 24 14:08:57.881842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:57.881808 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:57.909619 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:57.909562 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:57.910026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:57.909881 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" podUID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" containerName="payload-processing" containerID="cri-o://cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73" gracePeriod=30 Jul 24 14:08:58.163728 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.163696 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:58.193933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.193898 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:58.240435 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240402 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56nbs\" (UniqueName: \"kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs\") pod \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " Jul 24 14:08:58.240652 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240451 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca\") pod \"9d648725-a236-44ca-9463-eb7d4b06fa3b\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " Jul 24 14:08:58.240652 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240484 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls\") pod \"9d648725-a236-44ca-9463-eb7d4b06fa3b\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " Jul 24 14:08:58.240652 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240565 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8qjx\" (UniqueName: \"kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx\") pod \"9d648725-a236-44ca-9463-eb7d4b06fa3b\" (UID: \"9d648725-a236-44ca-9463-eb7d4b06fa3b\") " Jul 24 14:08:58.240839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240657 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume\") pod \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\" (UID: \"8e1af8d7-359e-4bc0-9d25-655823ce8a3c\") " Jul 24 14:08:58.240913 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.240885 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d648725-a236-44ca-9463-eb7d4b06fa3b" (UID: "9d648725-a236-44ca-9463-eb7d4b06fa3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:58.241149 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.241121 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e1af8d7-359e-4bc0-9d25-655823ce8a3c" (UID: "8e1af8d7-359e-4bc0-9d25-655823ce8a3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:58.242755 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.242727 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d648725-a236-44ca-9463-eb7d4b06fa3b" (UID: "9d648725-a236-44ca-9463-eb7d4b06fa3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:08:58.242755 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.242737 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs" (OuterVolumeSpecName: "kube-api-access-56nbs") pod "8e1af8d7-359e-4bc0-9d25-655823ce8a3c" (UID: "8e1af8d7-359e-4bc0-9d25-655823ce8a3c"). InnerVolumeSpecName "kube-api-access-56nbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:58.242893 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.242731 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx" (OuterVolumeSpecName: "kube-api-access-p8qjx") pod "9d648725-a236-44ca-9463-eb7d4b06fa3b" (UID: "9d648725-a236-44ca-9463-eb7d4b06fa3b"). InnerVolumeSpecName "kube-api-access-p8qjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:58.259655 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.259618 2581 generic.go:356] "Generic (PLEG): container finished" podID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" containerID="cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73" exitCode=0 Jul 24 14:08:58.259840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.259671 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" Jul 24 14:08:58.259840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.259710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" event={"ID":"8e1af8d7-359e-4bc0-9d25-655823ce8a3c","Type":"ContainerDied","Data":"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73"} Jul 24 14:08:58.259840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.259765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4" event={"ID":"8e1af8d7-359e-4bc0-9d25-655823ce8a3c","Type":"ContainerDied","Data":"7f93efe503e0980f2685b23b87f4bacb6e088475234d915de156b1e93abbffc1"} Jul 24 14:08:58.259840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.259802 2581 scope.go:117] "RemoveContainer" containerID="cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73" Jul 24 14:08:58.261116 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.261091 2581 generic.go:356] "Generic (PLEG): container finished" podID="9d648725-a236-44ca-9463-eb7d4b06fa3b" containerID="d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f" exitCode=0 Jul 24 14:08:58.261240 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.261147 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" Jul 24 14:08:58.261240 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.261148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" event={"ID":"9d648725-a236-44ca-9463-eb7d4b06fa3b","Type":"ContainerDied","Data":"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f"} Jul 24 14:08:58.261348 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.261255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4" event={"ID":"9d648725-a236-44ca-9463-eb7d4b06fa3b","Type":"ContainerDied","Data":"a6e58765cca79ac79ab980f825b4ee990accf9ffa877b34c637bb3f80dc36b3c"} Jul 24 14:08:58.270893 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.270869 2581 scope.go:117] "RemoveContainer" containerID="cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73" Jul 24 14:08:58.271284 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:58.271253 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73\": container with ID starting with cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73 not found: ID does not exist" containerID="cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73" Jul 24 14:08:58.271393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.271285 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73"} err="failed to get container status \"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73\": rpc error: code = NotFound desc = could not find container \"cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73\": container with ID starting with cf28c8d5766b771f0390099d395d01b70b2d481e5775ea58ecf1ef794d89bf73 not found: ID does not exist" Jul 24 14:08:58.271393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.271306 2581 scope.go:117] "RemoveContainer" containerID="d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f" Jul 24 14:08:58.280071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.280051 2581 scope.go:117] "RemoveContainer" containerID="d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f" Jul 24 14:08:58.280424 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:08:58.280404 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f\": container with ID starting with d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f not found: ID does not exist" containerID="d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f" Jul 24 14:08:58.280510 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.280432 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f"} err="failed to get container status \"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f\": rpc error: code = NotFound desc = could not find container \"d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f\": container with ID starting with d9bc3dccdb54d85025c4016fa040248ed044f08f291e677766ce119422e3d74f not found: ID does not exist" Jul 24 14:08:58.284807 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.284780 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:58.288507 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.288478 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-388084cb-7b5c586d68-988b4"] Jul 24 14:08:58.298183 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.298150 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:58.302712 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.302682 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-388084cb-5fdcd7d4c4-f27g4"] Jul 24 14:08:58.337366 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.337331 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:58.341458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.341423 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8qjx\" (UniqueName: \"kubernetes.io/projected/9d648725-a236-44ca-9463-eb7d4b06fa3b-kube-api-access-p8qjx\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.341458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.341453 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.341458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.341466 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56nbs\" (UniqueName: \"kubernetes.io/projected/8e1af8d7-359e-4bc0-9d25-655823ce8a3c-kube-api-access-56nbs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.341691 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.341475 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d648725-a236-44ca-9463-eb7d4b06fa3b-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.341691 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.341485 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d648725-a236-44ca-9463-eb7d4b06fa3b-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.402499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.402408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:08:58.922138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.922093 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" path="/var/lib/kubelet/pods/8e1af8d7-359e-4bc0-9d25-655823ce8a3c/volumes" Jul 24 14:08:58.922660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:08:58.922642 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d648725-a236-44ca-9463-eb7d4b06fa3b" path="/var/lib/kubelet/pods/9d648725-a236-44ca-9463-eb7d4b06fa3b/volumes" Jul 24 14:09:02.496318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:02.496279 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb"] Jul 24 14:09:02.498851 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:02.498821 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-388084cb-mmjrb"] Jul 24 14:09:02.504217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:02.504183 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:02.915958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:02.915931 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea8fd77-e757-4ede-af8a-632cd0ebce58" path="/var/lib/kubelet/pods/6ea8fd77-e757-4ede-af8a-632cd0ebce58/volumes" Jul 24 14:09:03.801817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:03.801769 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:09:03.802336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:03.802043 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" containerID="cri-o://fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788" gracePeriod=30 Jul 24 14:09:03.826724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:03.826688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:04.084941 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.084854 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:04.318440 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.318407 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:04.608372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608337 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:04.608738 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608724 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:09:04.608738 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608738 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608750 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea8fd77-e757-4ede-af8a-632cd0ebce58" containerName="revoke-keys" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608755 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea8fd77-e757-4ede-af8a-632cd0ebce58" containerName="revoke-keys" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608773 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d648725-a236-44ca-9463-eb7d4b06fa3b" containerName="maas-api" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608779 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d648725-a236-44ca-9463-eb7d4b06fa3b" containerName="maas-api" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608787 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" containerName="payload-processing" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608792 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" containerName="payload-processing" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608857 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18f3a59-37dc-48b1-b44c-90638c7403db" containerName="istio-proxy" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608870 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d648725-a236-44ca-9463-eb7d4b06fa3b" containerName="maas-api" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608879 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e1af8d7-359e-4bc0-9d25-655823ce8a3c" containerName="payload-processing" Jul 24 14:09:04.608884 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.608890 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea8fd77-e757-4ede-af8a-632cd0ebce58" containerName="revoke-keys" Jul 24 14:09:04.614639 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.614599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.618397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.618371 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a487c2a7-openshift-default-dockercfg-wqhdj\"" Jul 24 14:09:04.624262 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.623904 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:04.631336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.631305 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:04.701073 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxhl\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701073 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.701530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.701497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.802852 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.802808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.802852 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.802854 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.802914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.802962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxhl\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803324 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803374 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803530 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803670 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.803976 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.803934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.804240 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.804218 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.805561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.805536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.806401 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.806379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.810965 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.810940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.811089 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.811064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxhl\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl\") pod \"e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:04.928249 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:04.928211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:05.113371 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.113345 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:05.115610 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:09:05.115550 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4580e813_0455_4d36_83dd_62edfe6a1ea9.slice/crio-3757d2525a21f36eec2af277f0b2431290447ad1ca8a03650dc82a27e98ec7c6 WatchSource:0}: Error finding container 3757d2525a21f36eec2af277f0b2431290447ad1ca8a03650dc82a27e98ec7c6: Status 404 returned error can't find the container with id 3757d2525a21f36eec2af277f0b2431290447ad1ca8a03650dc82a27e98ec7c6 Jul 24 14:09:05.117961 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.117920 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:05.118084 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.118013 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:05.118084 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.118059 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:05.244350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.244318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:05.303012 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.302970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerStarted","Data":"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0"} Jul 24 14:09:05.303229 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.303117 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerStarted","Data":"3757d2525a21f36eec2af277f0b2431290447ad1ca8a03650dc82a27e98ec7c6"} Jul 24 14:09:05.323277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.323184 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podStartSLOduration=1.3231632420000001 podStartE2EDuration="1.323163242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:09:05.32261773 +0000 UTC m=+1838.919181786" watchObservedRunningTime="2026-07-24 14:09:05.323163242 +0000 UTC m=+1838.919727298" Jul 24 14:09:05.929128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:05.929089 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:06.688042 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.688007 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:06.776855 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.776820 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:06.780975 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.780948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:06.783576 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.783524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:06.783576 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.783559 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a487c2a7-serving-cert\"" Jul 24 14:09:06.783835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.783647 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:09:06.790341 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.790263 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:06.881919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.881884 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:06.927267 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.927234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:06.927267 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.927271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:06.927551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.927296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fch\" (UniqueName: \"kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:06.934464 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:06.934433 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:07.028024 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.027930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.028188 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.028102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.028188 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.028127 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fch\" (UniqueName: \"kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.028274 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:07.028242 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-serving-cert: secret "maas-api-e2e-derive-a487c2a7-serving-cert" not found Jul 24 14:09:07.028319 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:07.028313 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls podName:8ff73356-f5e9-4b08-a68f-b2eb5de5a718 nodeName:}" failed. No retries permitted until 2026-07-24 14:09:07.528291405 +0000 UTC m=+1841.124855443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls") pod "maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" (UID: "8ff73356-f5e9-4b08-a68f-b2eb5de5a718") : secret "maas-api-e2e-derive-a487c2a7-serving-cert" not found Jul 24 14:09:07.028813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.028793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.037632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.037579 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fch\" (UniqueName: \"kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.103162 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.103124 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:07.314208 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.314125 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:07.315268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.315246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:07.486381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.486350 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:07.533071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.533034 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.535510 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.535474 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") pod \"maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.584659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.584544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:07.617555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.617517 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:07.621480 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.621452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.624547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.624489 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a487c2a7\"" Jul 24 14:09:07.624782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.624607 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a487c2a7-dockercfg-kpk6n\"" Jul 24 14:09:07.633397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.633357 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:07.696263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.696224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:07.734813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.734776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.735032 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.734839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc7r\" (UniqueName: \"kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.836229 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.836151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.836229 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.836197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc7r\" (UniqueName: \"kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.836969 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.836949 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.845306 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.845273 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc7r\" (UniqueName: \"kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r\") pod \"payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:07.854095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.853699 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:07.858567 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:09:07.858530 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff73356_f5e9_4b08_a68f_b2eb5de5a718.slice/crio-aa0da7c51abbca1f58add975cc48ff694fd01bb02b266eb5857bd31010dd62de WatchSource:0}: Error finding container aa0da7c51abbca1f58add975cc48ff694fd01bb02b266eb5857bd31010dd62de: Status 404 returned error can't find the container with id aa0da7c51abbca1f58add975cc48ff694fd01bb02b266eb5857bd31010dd62de Jul 24 14:09:07.935094 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:07.935061 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:08.095226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.095134 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:08.098530 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:09:08.098487 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee44231_f788_4381_b66d_522a1bc3d103.slice/crio-f4d57a5bda38d56a54782e0f68d831b3c9ac6dae073c684dd8f98b63f1ff3193 WatchSource:0}: Error finding container f4d57a5bda38d56a54782e0f68d831b3c9ac6dae073c684dd8f98b63f1ff3193: Status 404 returned error can't find the container with id f4d57a5bda38d56a54782e0f68d831b3c9ac6dae073c684dd8f98b63f1ff3193 Jul 24 14:09:08.319808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.319774 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" event={"ID":"8ff73356-f5e9-4b08-a68f-b2eb5de5a718","Type":"ContainerStarted","Data":"aa0da7c51abbca1f58add975cc48ff694fd01bb02b266eb5857bd31010dd62de"} Jul 24 14:09:08.321845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.321809 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" event={"ID":"dee44231-f788-4381-b66d-522a1bc3d103","Type":"ContainerStarted","Data":"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe"} Jul 24 14:09:08.321993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.321935 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:08.321993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.321981 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" event={"ID":"dee44231-f788-4381-b66d-522a1bc3d103","Type":"ContainerStarted","Data":"f4d57a5bda38d56a54782e0f68d831b3c9ac6dae073c684dd8f98b63f1ff3193"} Jul 24 14:09:08.348420 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:08.348307 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" podStartSLOduration=1.3482902860000001 podStartE2EDuration="1.348290286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:09:08.347091085 +0000 UTC m=+1841.943655140" watchObservedRunningTime="2026-07-24 14:09:08.348290286 +0000 UTC m=+1841.944854341" Jul 24 14:09:09.068426 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.068344 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:09:09.150187 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150151 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150187 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150189 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150248 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150273 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vpg6\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150353 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150374 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150404 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150434 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data\") pod \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\" (UID: \"62c7af01-7a55-46c2-9e95-ba18c792d7dc\") " Jul 24 14:09:09.150842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150690 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:09.150842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150708 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:09.150956 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150933 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:09.151013 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.150990 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:09.151603 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.151564 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data" (OuterVolumeSpecName: "istio-data") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:09.152641 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.152570 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:09.152771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.152717 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6" (OuterVolumeSpecName: "kube-api-access-9vpg6") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "kube-api-access-9vpg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:09.152897 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.152880 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:09:09.152987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.152973 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "62c7af01-7a55-46c2-9e95-ba18c792d7dc" (UID: "62c7af01-7a55-46c2-9e95-ba18c792d7dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:09.252074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252028 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252066 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252075 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252082 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252089 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252387 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252097 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c7af01-7a55-46c2-9e95-ba18c792d7dc-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252387 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252106 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpg6\" (UniqueName: \"kubernetes.io/projected/62c7af01-7a55-46c2-9e95-ba18c792d7dc-kube-api-access-9vpg6\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252387 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252114 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.252387 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.252121 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c7af01-7a55-46c2-9e95-ba18c792d7dc-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:09.328083 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.327996 2581 generic.go:356] "Generic (PLEG): container finished" podID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerID="fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788" exitCode=0 Jul 24 14:09:09.328083 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.328042 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" Jul 24 14:09:09.328083 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.328072 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerDied","Data":"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788"} Jul 24 14:09:09.328358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.328134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst" event={"ID":"62c7af01-7a55-46c2-9e95-ba18c792d7dc","Type":"ContainerDied","Data":"dbce3191db58f05973e95111dcea366c8acc7e36e1edee32379fe822fa7a7820"} Jul 24 14:09:09.328358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.328169 2581 scope.go:117] "RemoveContainer" containerID="fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788" Jul 24 14:09:09.329806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.329782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" event={"ID":"8ff73356-f5e9-4b08-a68f-b2eb5de5a718","Type":"ContainerStarted","Data":"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2"} Jul 24 14:09:09.329935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.329919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:09.338116 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.338097 2581 scope.go:117] "RemoveContainer" containerID="d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e" Jul 24 14:09:09.346784 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.346762 2581 scope.go:117] "RemoveContainer" containerID="fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788" Jul 24 14:09:09.347109 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:09.347088 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788\": container with ID starting with fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788 not found: ID does not exist" containerID="fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788" Jul 24 14:09:09.347160 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.347122 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788"} err="failed to get container status \"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788\": rpc error: code = NotFound desc = could not find container \"fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788\": container with ID starting with fb1dafd2f79c4849bafd4e5da25eb830ad7be23bd0bf13210fefdf4e5ab62788 not found: ID does not exist" Jul 24 14:09:09.347160 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.347142 2581 scope.go:117] "RemoveContainer" containerID="d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e" Jul 24 14:09:09.347409 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:09.347391 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e\": container with ID starting with d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e not found: ID does not exist" containerID="d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e" Jul 24 14:09:09.347462 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.347418 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e"} err="failed to get container status \"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e\": rpc error: code = NotFound desc = could not find container \"d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e\": container with ID starting with d50bb9d1aae7c0b4917e157b171d47de7fe8a91bf7e1eba25847425deaccea5e not found: ID does not exist" Jul 24 14:09:09.349074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.349020 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" podStartSLOduration=2.959275518 podStartE2EDuration="3.349005795s" totalImagesPullingTime="389.730277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:09:09.348575576 +0000 UTC m=+1842.945139635" watchObservedRunningTime="2026-07-24 14:09:09.349005795 +0000 UTC m=+1842.945569850" Jul 24 14:09:09.362646 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.362610 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:09:09.370050 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:09.370020 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-388084cb-openshift-default-574f98d-wmkst"] Jul 24 14:09:10.916914 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:10.916875 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" path="/var/lib/kubelet/pods/62c7af01-7a55-46c2-9e95-ba18c792d7dc/volumes" Jul 24 14:09:15.342500 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:15.342468 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:19.333093 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:19.333062 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:21.399175 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.399143 2581 generic.go:356] "Generic (PLEG): container finished" podID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerID="43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0" exitCode=0 Jul 24 14:09:21.399700 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.399211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerDied","Data":"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0"} Jul 24 14:09:21.399835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.399817 2581 scope.go:117] "RemoveContainer" containerID="43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0" Jul 24 14:09:21.400809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.400782 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:21.400922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.400851 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:21.400922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:21.400883 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:09:22.314661 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:22.314611 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:22.406207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:22.406168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerStarted","Data":"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9"} Jul 24 14:09:22.928972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:22.928937 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:22.930242 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:22.930215 2581 patch_prober.go:28] interesting pod/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 14:09:22.930375 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:22.930268 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 14:09:23.933809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:23.933771 2581 patch_prober.go:28] interesting pod/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 14:09:23.934230 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:23.933833 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 14:09:24.928824 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:24.928781 2581 patch_prober.go:28] interesting pod/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 14:09:24.929013 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:24.928839 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 14:09:25.929559 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:25.929517 2581 patch_prober.go:28] interesting pod/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 14:09:25.929993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:25.929576 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 14:09:26.933510 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:26.933479 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:26.933933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:26.933811 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:26.934737 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:26.934717 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:31.621714 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.621675 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44"] Jul 24 14:09:31.622140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622065 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.622140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622076 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.622140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622085 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.622140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622090 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.622293 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622163 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.622293 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.622174 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c7af01-7a55-46c2-9e95-ba18c792d7dc" containerName="istio-proxy" Jul 24 14:09:31.625199 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.625182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:31.627781 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.627761 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:09:31.633721 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.633649 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44"] Jul 24 14:09:31.662882 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.662841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44\" (UID: \"272ec6aa-7899-4c29-b016-ad15e0556b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:31.764008 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.763971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44\" (UID: \"272ec6aa-7899-4c29-b016-ad15e0556b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:31.764572 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.764554 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44\" (UID: \"272ec6aa-7899-4c29-b016-ad15e0556b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:31.938449 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:31.938412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:32.298165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:32.298127 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44"] Jul 24 14:09:32.300866 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:09:32.300837 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272ec6aa_7899_4c29_b016_ad15e0556b68.slice/crio-0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689 WatchSource:0}: Error finding container 0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689: Status 404 returned error can't find the container with id 0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689 Jul 24 14:09:32.457333 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:32.457298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" event={"ID":"272ec6aa-7899-4c29-b016-ad15e0556b68","Type":"ContainerStarted","Data":"0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689"} Jul 24 14:09:33.464724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:33.464686 2581 generic.go:356] "Generic (PLEG): container finished" podID="272ec6aa-7899-4c29-b016-ad15e0556b68" containerID="f2bf74170c4e00df8f07b7d5f9195909d5a057509651ab8d2cb8b5277b8fa704" exitCode=0 Jul 24 14:09:33.465129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:33.464779 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" event={"ID":"272ec6aa-7899-4c29-b016-ad15e0556b68","Type":"ContainerDied","Data":"f2bf74170c4e00df8f07b7d5f9195909d5a057509651ab8d2cb8b5277b8fa704"} Jul 24 14:09:34.614938 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:34.614908 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:34.691891 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:34.691852 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca\") pod \"272ec6aa-7899-4c29-b016-ad15e0556b68\" (UID: \"272ec6aa-7899-4c29-b016-ad15e0556b68\") " Jul 24 14:09:34.692239 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:34.692213 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "272ec6aa-7899-4c29-b016-ad15e0556b68" (UID: "272ec6aa-7899-4c29-b016-ad15e0556b68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:34.793040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:34.792933 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272ec6aa-7899-4c29-b016-ad15e0556b68-maas-api-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:35.476525 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:35.476496 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" Jul 24 14:09:35.476525 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:35.476512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44" event={"ID":"272ec6aa-7899-4c29-b016-ad15e0556b68","Type":"ContainerDied","Data":"0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689"} Jul 24 14:09:35.476775 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:35.476554 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ebad3c72729bafc0a753f068ba3d05f1f35e434a98a668c1796e1e7fe813689" Jul 24 14:09:42.130609 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.129888 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:42.130609 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.130142 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" podUID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" containerName="maas-api" containerID="cri-o://9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2" gracePeriod=30 Jul 24 14:09:42.157839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.157792 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:42.187561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.187512 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:42.187826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.187798 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" podUID="dee44231-f788-4381-b66d-522a1bc3d103" containerName="payload-processing" containerID="cri-o://23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe" gracePeriod=30 Jul 24 14:09:42.258432 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.258397 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:42.359809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.359762 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:42.462839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.462803 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:42.478792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.478758 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:42.512157 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.512118 2581 generic.go:356] "Generic (PLEG): container finished" podID="dee44231-f788-4381-b66d-522a1bc3d103" containerID="23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe" exitCode=0 Jul 24 14:09:42.512395 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.512176 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" Jul 24 14:09:42.512395 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.512211 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" event={"ID":"dee44231-f788-4381-b66d-522a1bc3d103","Type":"ContainerDied","Data":"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe"} Jul 24 14:09:42.512395 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.512288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2" event={"ID":"dee44231-f788-4381-b66d-522a1bc3d103","Type":"ContainerDied","Data":"f4d57a5bda38d56a54782e0f68d831b3c9ac6dae073c684dd8f98b63f1ff3193"} Jul 24 14:09:42.512395 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.512329 2581 scope.go:117] "RemoveContainer" containerID="23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe" Jul 24 14:09:42.513905 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.513810 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" containerID="9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2" exitCode=0 Jul 24 14:09:42.513905 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.513845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" event={"ID":"8ff73356-f5e9-4b08-a68f-b2eb5de5a718","Type":"ContainerDied","Data":"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2"} Jul 24 14:09:42.513905 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.513872 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" Jul 24 14:09:42.513905 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.513895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm" event={"ID":"8ff73356-f5e9-4b08-a68f-b2eb5de5a718","Type":"ContainerDied","Data":"aa0da7c51abbca1f58add975cc48ff694fd01bb02b266eb5857bd31010dd62de"} Jul 24 14:09:42.521907 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.521878 2581 scope.go:117] "RemoveContainer" containerID="23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe" Jul 24 14:09:42.522275 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:42.522257 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe\": container with ID starting with 23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe not found: ID does not exist" containerID="23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe" Jul 24 14:09:42.522354 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.522284 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe"} err="failed to get container status \"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe\": rpc error: code = NotFound desc = could not find container \"23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe\": container with ID starting with 23ed711654bc1a13a858ffe4516f25d8f45cee34057f0714571bb41afd0ad5fe not found: ID does not exist" Jul 24 14:09:42.522354 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.522301 2581 scope.go:117] "RemoveContainer" containerID="9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2" Jul 24 14:09:42.531043 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.531021 2581 scope.go:117] "RemoveContainer" containerID="9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2" Jul 24 14:09:42.531408 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:42.531386 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2\": container with ID starting with 9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2 not found: ID does not exist" containerID="9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2" Jul 24 14:09:42.531454 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.531419 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2"} err="failed to get container status \"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2\": rpc error: code = NotFound desc = could not find container \"9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2\": container with ID starting with 9896c811347340500d96f81e5a32c352edd1631d70ea409c8aa2a3f1b0cc81e2 not found: ID does not exist" Jul 24 14:09:42.560808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.560771 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca\") pod \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " Jul 24 14:09:42.560808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.560809 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrc7r\" (UniqueName: \"kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r\") pod \"dee44231-f788-4381-b66d-522a1bc3d103\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " Jul 24 14:09:42.561039 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.560861 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") pod \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " Jul 24 14:09:42.561039 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.560890 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume\") pod \"dee44231-f788-4381-b66d-522a1bc3d103\" (UID: \"dee44231-f788-4381-b66d-522a1bc3d103\") " Jul 24 14:09:42.561039 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.560920 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4fch\" (UniqueName: \"kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch\") pod \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\" (UID: \"8ff73356-f5e9-4b08-a68f-b2eb5de5a718\") " Jul 24 14:09:42.561266 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.561242 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ff73356-f5e9-4b08-a68f-b2eb5de5a718" (UID: "8ff73356-f5e9-4b08-a68f-b2eb5de5a718"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:42.561371 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.561335 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dee44231-f788-4381-b66d-522a1bc3d103" (UID: "dee44231-f788-4381-b66d-522a1bc3d103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:42.563269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.563238 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ff73356-f5e9-4b08-a68f-b2eb5de5a718" (UID: "8ff73356-f5e9-4b08-a68f-b2eb5de5a718"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:09:42.563269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.563255 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r" (OuterVolumeSpecName: "kube-api-access-xrc7r") pod "dee44231-f788-4381-b66d-522a1bc3d103" (UID: "dee44231-f788-4381-b66d-522a1bc3d103"). InnerVolumeSpecName "kube-api-access-xrc7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:42.563467 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.563324 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch" (OuterVolumeSpecName: "kube-api-access-c4fch") pod "8ff73356-f5e9-4b08-a68f-b2eb5de5a718" (UID: "8ff73356-f5e9-4b08-a68f-b2eb5de5a718"). InnerVolumeSpecName "kube-api-access-c4fch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:42.662430 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.662338 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.662430 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.662368 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrc7r\" (UniqueName: \"kubernetes.io/projected/dee44231-f788-4381-b66d-522a1bc3d103-kube-api-access-xrc7r\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.662430 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.662379 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.662430 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.662389 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee44231-f788-4381-b66d-522a1bc3d103-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.662430 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.662397 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4fch\" (UniqueName: \"kubernetes.io/projected/8ff73356-f5e9-4b08-a68f-b2eb5de5a718-kube-api-access-c4fch\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.841761 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.841718 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:42.847041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.847009 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a487c2a7-774b94bfbb-s65b2"] Jul 24 14:09:42.857529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.857495 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:42.861794 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.861763 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a487c2a7-7c655b988d-j78rm"] Jul 24 14:09:42.916298 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.916269 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" path="/var/lib/kubelet/pods/8ff73356-f5e9-4b08-a68f-b2eb5de5a718/volumes" Jul 24 14:09:42.916688 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:42.916674 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee44231-f788-4381-b66d-522a1bc3d103" path="/var/lib/kubelet/pods/dee44231-f788-4381-b66d-522a1bc3d103/volumes" Jul 24 14:09:46.691757 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.691721 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44"] Jul 24 14:09:46.694284 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.694251 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a487c2a7-l9b44"] Jul 24 14:09:46.717922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.717888 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:46.839277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.839235 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:46.839647 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.839564 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" containerID="cri-o://6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9" gracePeriod=30 Jul 24 14:09:46.916136 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.916098 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272ec6aa-7899-4c29-b016-ad15e0556b68" path="/var/lib/kubelet/pods/272ec6aa-7899-4c29-b016-ad15e0556b68/volumes" Jul 24 14:09:46.926824 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:46.926787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:47.293099 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:47.293068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:47.652286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:47.652197 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:47.749464 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:47.749434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:47.851215 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:47.851183 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:49.510888 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:49.510804 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:52.104057 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.103974 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:52.248413 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248377 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248435 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248485 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248617 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcxhl\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248841 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248661 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248841 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248686 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248841 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248758 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248841 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248790 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.248841 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.248820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs\") pod \"4580e813-0455-4d36-83dd-62edfe6a1ea9\" (UID: \"4580e813-0455-4d36-83dd-62edfe6a1ea9\") " Jul 24 14:09:52.249343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.249069 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:52.249343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.249301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:52.249343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.249313 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:52.249535 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.249409 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:52.249535 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.249425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:52.250924 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.250895 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:52.250924 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.250914 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:09:52.251185 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.251156 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:09:52.251185 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.251171 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl" (OuterVolumeSpecName: "kube-api-access-kcxhl") pod "4580e813-0455-4d36-83dd-62edfe6a1ea9" (UID: "4580e813-0455-4d36-83dd-62edfe6a1ea9"). InnerVolumeSpecName "kube-api-access-kcxhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:52.350422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350368 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350413 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcxhl\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-kube-api-access-kcxhl\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350424 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4580e813-0455-4d36-83dd-62edfe6a1ea9-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350434 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350442 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350450 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350458 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350466 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.350769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.350473 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4580e813-0455-4d36-83dd-62edfe6a1ea9-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:09:52.569217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.569176 2581 generic.go:356] "Generic (PLEG): container finished" podID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerID="6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9" exitCode=0 Jul 24 14:09:52.569427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.569235 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" Jul 24 14:09:52.569427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.569265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerDied","Data":"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9"} Jul 24 14:09:52.569427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.569316 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh" event={"ID":"4580e813-0455-4d36-83dd-62edfe6a1ea9","Type":"ContainerDied","Data":"3757d2525a21f36eec2af277f0b2431290447ad1ca8a03650dc82a27e98ec7c6"} Jul 24 14:09:52.569427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.569341 2581 scope.go:117] "RemoveContainer" containerID="6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9" Jul 24 14:09:52.579204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.579183 2581 scope.go:117] "RemoveContainer" containerID="43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0" Jul 24 14:09:52.588446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.588421 2581 scope.go:117] "RemoveContainer" containerID="6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9" Jul 24 14:09:52.588844 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:52.588823 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9\": container with ID starting with 6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9 not found: ID does not exist" containerID="6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9" Jul 24 14:09:52.588910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.588855 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9"} err="failed to get container status \"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9\": rpc error: code = NotFound desc = could not find container \"6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9\": container with ID starting with 6b3a7d33c4c0c712925502e6a4d9f4abff14b964bfb0f58878204143e5d47bd9 not found: ID does not exist" Jul 24 14:09:52.588910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.588874 2581 scope.go:117] "RemoveContainer" containerID="43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0" Jul 24 14:09:52.589195 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:09:52.589176 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0\": container with ID starting with 43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0 not found: ID does not exist" containerID="43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0" Jul 24 14:09:52.589246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.589201 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0"} err="failed to get container status \"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0\": rpc error: code = NotFound desc = could not find container \"43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0\": container with ID starting with 43fc37d680e0225e9fc94e5bba560f5964c6cae8f638cdf0b3d40fd7ce0dc8c0 not found: ID does not exist" Jul 24 14:09:52.591870 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.591841 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:52.598035 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.598002 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a487c2a7-openshift-default-7bffb854f6-5xmfh"] Jul 24 14:09:52.916499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:52.916466 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" path="/var/lib/kubelet/pods/4580e813-0455-4d36-83dd-62edfe6a1ea9/volumes" Jul 24 14:09:55.457843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:55.457805 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:55.559887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:55.559855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:55.658060 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:55.658020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:55.766383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:55.766289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:09:56.262551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:09:56.262522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:22.255456 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:22.255404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:28.998283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:28.998233 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:29.106078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:29.106047 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:29.198777 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:29.198745 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:45.216847 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:45.216809 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:51.002274 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:51.002243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:51.068996 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:51.068957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:51.144897 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:51.144864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:51.239097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:51.239067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:56.458740 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:56.458704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:10:56.518505 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:10:56.518472 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:01.807437 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:01.807401 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:02.741706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:02.741677 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:09.040379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:09.040342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:09.143076 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:09.143041 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:09.240490 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:09.240458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:09.340525 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:09.340439 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:10.169173 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:10.169135 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:10.270631 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:10.270548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:10.369921 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:10.369881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:12.087139 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:12.087088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:18.052842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:18.052750 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:18.163706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:18.163672 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:18.800277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:18.800247 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:25.478071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:25.478034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:25.573993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:25.573957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:25.740123 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:25.740042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:26.050641 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.050529 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:11:26.051096 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051079 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee44231-f788-4381-b66d-522a1bc3d103" containerName="payload-processing" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051098 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee44231-f788-4381-b66d-522a1bc3d103" containerName="payload-processing" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051119 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051128 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051139 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051147 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051168 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272ec6aa-7899-4c29-b016-ad15e0556b68" containerName="revoke-keys" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051176 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="272ec6aa-7899-4c29-b016-ad15e0556b68" containerName="revoke-keys" Jul 24 14:11:26.051195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051196 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" containerName="maas-api" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051204 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" containerName="maas-api" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051273 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051283 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee44231-f788-4381-b66d-522a1bc3d103" containerName="payload-processing" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051292 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff73356-f5e9-4b08-a68f-b2eb5de5a718" containerName="maas-api" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051301 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="272ec6aa-7899-4c29-b016-ad15e0556b68" containerName="revoke-keys" Jul 24 14:11:26.051477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.051310 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4580e813-0455-4d36-83dd-62edfe6a1ea9" containerName="istio-proxy" Jul 24 14:11:26.054700 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.054679 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.057233 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.057204 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50cb7407-openshift-default-dockercfg-bvb6n\"" Jul 24 14:11:26.057376 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.057247 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 14:11:26.074486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.074445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:11:26.079517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.079480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:26.158105 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158105 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.158514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.158405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz58r\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.183189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.183156 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:26.259491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz58r\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259985 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.259985 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.260100 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.259992 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.260100 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.260082 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.260268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.260245 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.260388 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.260369 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.260842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.260819 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.262128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.262098 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.262315 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.262294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.273651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.273618 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.274045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.274025 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz58r\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r\") pod \"e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.365850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.365753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:26.514557 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.514527 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:11:26.516574 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:11:26.516544 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4784e70_335b_4198_85a1_83fc4bc1dbf4.slice/crio-d6fe3edddbd07d9ac342d80eece3ad68d0db5b9ca7c7961b88def7fbbe746dce WatchSource:0}: Error finding container d6fe3edddbd07d9ac342d80eece3ad68d0db5b9ca7c7961b88def7fbbe746dce: Status 404 returned error can't find the container with id d6fe3edddbd07d9ac342d80eece3ad68d0db5b9ca7c7961b88def7fbbe746dce Jul 24 14:11:26.524699 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.519326 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:26.524699 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.519480 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:26.524699 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:26.519573 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:27.020455 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:27.020422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerStarted","Data":"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e"} Jul 24 14:11:27.020823 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:27.020473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerStarted","Data":"d6fe3edddbd07d9ac342d80eece3ad68d0db5b9ca7c7961b88def7fbbe746dce"} Jul 24 14:11:27.043539 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:27.043474 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podStartSLOduration=1.043459116 podStartE2EDuration="1.043459116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:27.042084601 +0000 UTC m=+1980.638648668" watchObservedRunningTime="2026-07-24 14:11:27.043459116 +0000 UTC m=+1980.640023171" Jul 24 14:11:27.366246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:27.366138 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:27.372128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:27.372096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:28.026299 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:28.026265 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:28.027195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:28.027175 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:29.118152 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.118115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:29.213197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.213148 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:29.220794 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.220762 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:29.224729 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.224697 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.227009 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.226979 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-50cb7407-serving-cert\"" Jul 24 14:11:29.227009 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.226981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:11:29.236482 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.236234 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:29.286268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.286230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.286473 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.286315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.286473 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.286349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.310323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.310282 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:29.387194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.387109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.387194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.387183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.387394 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.387223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.387394 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:11:29.387344 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-serving-cert: secret "maas-api-e2e-mt-50cb7407-serving-cert" not found Jul 24 14:11:29.387461 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:11:29.387413 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls podName:3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d nodeName:}" failed. No retries permitted until 2026-07-24 14:11:29.887392382 +0000 UTC m=+1983.483956429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls") pod "maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" (UID: "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d") : secret "maas-api-e2e-mt-50cb7407-serving-cert" not found Jul 24 14:11:29.387915 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.387892 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.395618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.395577 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.580306 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.580271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:29.891308 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.891276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:29.893722 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:29.893690 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") pod \"maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:30.040573 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.040527 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:30.045251 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.045220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.047476 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.047450 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-50cb7407\"" Jul 24 14:11:30.047671 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.047557 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-50cb7407-dockercfg-zfxh2\"" Jul 24 14:11:30.053747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.053715 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:30.084189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.084153 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:30.087735 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.087712 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.093466 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.093439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d58q\" (UniqueName: \"kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.093662 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.093516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.097530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.097502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:30.097719 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.097645 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:30.137553 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.137515 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:30.195023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.194991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjfj\" (UniqueName: \"kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.195204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.195117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d58q\" (UniqueName: \"kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.195412 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.195390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.195625 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.195577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.196255 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.196235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.202269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.202199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:30.206750 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.206694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d58q\" (UniqueName: \"kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q\") pod \"payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.296267 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.296231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.296447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.296293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjfj\" (UniqueName: \"kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.297194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.297175 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.297637 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.297557 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:30.300444 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:11:30.300417 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3825a6e4_85cc_4e53_b91a_bf29a5bb1d5d.slice/crio-60c980073add15a9fff269a69bdb680a6f7f9fd993a7b923539221ab6c1fa362 WatchSource:0}: Error finding container 60c980073add15a9fff269a69bdb680a6f7f9fd993a7b923539221ab6c1fa362: Status 404 returned error can't find the container with id 60c980073add15a9fff269a69bdb680a6f7f9fd993a7b923539221ab6c1fa362 Jul 24 14:11:30.310711 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.310685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjfj\" (UniqueName: \"kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj\") pod \"payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.358763 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.358730 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:30.399724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.399631 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:30.523970 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.523941 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:30.527315 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:11:30.526050 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66cfea2_86c3_4f83_8bf6_f876eb10a7da.slice/crio-5c1c447a251214dae0b3f9c3db9b9fb1416898a111dfca647fa21c2ace7ac11c WatchSource:0}: Error finding container 5c1c447a251214dae0b3f9c3db9b9fb1416898a111dfca647fa21c2ace7ac11c: Status 404 returned error can't find the container with id 5c1c447a251214dae0b3f9c3db9b9fb1416898a111dfca647fa21c2ace7ac11c Jul 24 14:11:30.566075 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:30.566043 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:30.571818 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:11:30.571792 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d74fc7f_56d1_40ef_8041_f60ba05668a4.slice/crio-01971a6a76df5c7ef2ddfa92b8e84a7eb03d337570a726d262238b917e94f47e WatchSource:0}: Error finding container 01971a6a76df5c7ef2ddfa92b8e84a7eb03d337570a726d262238b917e94f47e: Status 404 returned error can't find the container with id 01971a6a76df5c7ef2ddfa92b8e84a7eb03d337570a726d262238b917e94f47e Jul 24 14:11:31.045471 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.045371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" event={"ID":"8d74fc7f-56d1-40ef-8041-f60ba05668a4","Type":"ContainerStarted","Data":"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3"} Jul 24 14:11:31.045471 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.045438 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:31.045471 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.045456 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" event={"ID":"8d74fc7f-56d1-40ef-8041-f60ba05668a4","Type":"ContainerStarted","Data":"01971a6a76df5c7ef2ddfa92b8e84a7eb03d337570a726d262238b917e94f47e"} Jul 24 14:11:31.047033 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.047001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" event={"ID":"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d","Type":"ContainerStarted","Data":"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7"} Jul 24 14:11:31.047172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.047054 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" event={"ID":"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d","Type":"ContainerStarted","Data":"60c980073add15a9fff269a69bdb680a6f7f9fd993a7b923539221ab6c1fa362"} Jul 24 14:11:31.047172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.047080 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:31.048575 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.048555 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" event={"ID":"b66cfea2-86c3-4f83-8bf6-f876eb10a7da","Type":"ContainerStarted","Data":"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194"} Jul 24 14:11:31.048702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.048620 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" event={"ID":"b66cfea2-86c3-4f83-8bf6-f876eb10a7da","Type":"ContainerStarted","Data":"5c1c447a251214dae0b3f9c3db9b9fb1416898a111dfca647fa21c2ace7ac11c"} Jul 24 14:11:31.048702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.048661 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:31.062150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.062102 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" podStartSLOduration=1.062088513 podStartE2EDuration="1.062088513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:31.061758964 +0000 UTC m=+1984.658323019" watchObservedRunningTime="2026-07-24 14:11:31.062088513 +0000 UTC m=+1984.658652567" Jul 24 14:11:31.079317 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.079266 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" podStartSLOduration=1.639902337 podStartE2EDuration="2.079249348s" totalImagesPullingTime="439.347011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:31.078261558 +0000 UTC m=+1984.674825628" watchObservedRunningTime="2026-07-24 14:11:31.079249348 +0000 UTC m=+1984.675813405" Jul 24 14:11:31.096729 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:31.096663 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" podStartSLOduration=1.096638189 podStartE2EDuration="1.096638189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:31.094000873 +0000 UTC m=+1984.690564927" watchObservedRunningTime="2026-07-24 14:11:31.096638189 +0000 UTC m=+1984.693202248" Jul 24 14:11:33.571831 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:33.571790 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:34.320181 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:34.320143 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:34.403438 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:34.403404 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:34.797188 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:34.797154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:37.059632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:37.059607 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:40.125370 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.125333 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:40.562205 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.562158 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:40.662055 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.662015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:40.925708 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.925664 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk"] Jul 24 14:11:40.930550 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.930525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:40.932878 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.932856 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:11:40.936837 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.936809 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk"] Jul 24 14:11:40.995198 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:40.995079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk\" (UID: \"ba5846e7-621b-40e8-ad7c-741d13587cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:41.095883 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:41.095845 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk\" (UID: \"ba5846e7-621b-40e8-ad7c-741d13587cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:41.096556 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:41.096529 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk\" (UID: \"ba5846e7-621b-40e8-ad7c-741d13587cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:41.244844 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:41.244812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:41.602486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:41.602255 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk"] Jul 24 14:11:41.605232 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:11:41.605193 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5846e7_621b_40e8_ad7c_741d13587cc7.slice/crio-40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e WatchSource:0}: Error finding container 40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e: Status 404 returned error can't find the container with id 40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e Jul 24 14:11:42.056779 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:42.056748 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:42.057193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:42.057166 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:42.123092 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:42.123053 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" event={"ID":"ba5846e7-621b-40e8-ad7c-741d13587cc7","Type":"ContainerStarted","Data":"1df6521bba3c358cea169b0cee07d1bf2cf9994eebea7cd4df3b67a7d2f34b05"} Jul 24 14:11:42.123380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:42.123358 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" event={"ID":"ba5846e7-621b-40e8-ad7c-741d13587cc7","Type":"ContainerStarted","Data":"40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e"} Jul 24 14:11:42.143821 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:42.143747 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" podStartSLOduration=2.143720964 podStartE2EDuration="2.143720964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:42.141177047 +0000 UTC m=+1995.737741102" watchObservedRunningTime="2026-07-24 14:11:42.143720964 +0000 UTC m=+1995.740285020" Jul 24 14:11:43.128915 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:43.128882 2581 generic.go:356] "Generic (PLEG): container finished" podID="ba5846e7-621b-40e8-ad7c-741d13587cc7" containerID="1df6521bba3c358cea169b0cee07d1bf2cf9994eebea7cd4df3b67a7d2f34b05" exitCode=0 Jul 24 14:11:43.129336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:43.128959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" event={"ID":"ba5846e7-621b-40e8-ad7c-741d13587cc7","Type":"ContainerDied","Data":"1df6521bba3c358cea169b0cee07d1bf2cf9994eebea7cd4df3b67a7d2f34b05"} Jul 24 14:11:44.067790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.067753 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:11:44.070023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.067831 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:11:44.135468 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.135439 2581 generic.go:356] "Generic (PLEG): container finished" podID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerID="d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e" exitCode=0 Jul 24 14:11:44.135903 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.135515 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerDied","Data":"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e"} Jul 24 14:11:44.136274 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.136010 2581 scope.go:117] "RemoveContainer" containerID="d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e" Jul 24 14:11:44.137019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.136993 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:44.137112 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.137056 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:44.137112 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.137087 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:11:44.306127 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.306086 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:44.423298 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.423267 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca\") pod \"ba5846e7-621b-40e8-ad7c-741d13587cc7\" (UID: \"ba5846e7-621b-40e8-ad7c-741d13587cc7\") " Jul 24 14:11:44.423665 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.423644 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba5846e7-621b-40e8-ad7c-741d13587cc7" (UID: "ba5846e7-621b-40e8-ad7c-741d13587cc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:11:44.524380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:44.524348 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba5846e7-621b-40e8-ad7c-741d13587cc7-maas-api-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:45.141676 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.141643 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerStarted","Data":"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813"} Jul 24 14:11:45.142997 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.142974 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" Jul 24 14:11:45.143131 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.143004 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk" event={"ID":"ba5846e7-621b-40e8-ad7c-741d13587cc7","Type":"ContainerDied","Data":"40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e"} Jul 24 14:11:45.143131 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.143046 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a5830266c74ae6016f2106da50dcfa13fbe5754824509d7c4a32fb4f576e8e" Jul 24 14:11:45.366951 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.366846 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:45.367172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.367142 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 14:11:45.367248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:45.367222 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 14:11:46.366485 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:46.366443 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 14:11:46.367027 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:46.366510 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 14:11:47.367060 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:47.367024 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 14:11:47.367685 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:47.367652 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 14:11:48.367033 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:48.366999 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 14:11:48.367319 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:48.367291 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 14:11:49.369974 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:49.369942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:49.370373 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:49.370216 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:49.371079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:49.371060 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:11:51.226021 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.225982 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:51.226491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.226300 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" podUID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" containerName="maas-api" containerID="cri-o://cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7" gracePeriod=30 Jul 24 14:11:51.260844 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.260814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:51.268537 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.268490 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:51.269056 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.269019 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" podUID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" containerName="payload-processing" containerID="cri-o://b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194" gracePeriod=30 Jul 24 14:11:51.276334 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.276303 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:51.276573 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.276548 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" podUID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" containerName="payload-pre-processing" containerID="cri-o://fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3" gracePeriod=30 Jul 24 14:11:51.364223 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.364186 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:51.551332 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.551305 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:51.565222 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.565193 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:51.584121 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.583835 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:51.686011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.685971 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca\") pod \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " Jul 24 14:11:51.686011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686010 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume\") pod \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " Jul 24 14:11:51.686238 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686075 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664\") pod \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " Jul 24 14:11:51.686238 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686179 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d58q\" (UniqueName: \"kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q\") pod \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\" (UID: \"b66cfea2-86c3-4f83-8bf6-f876eb10a7da\") " Jul 24 14:11:51.686238 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686208 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kjfj\" (UniqueName: \"kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj\") pod \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " Jul 24 14:11:51.686402 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686241 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume\") pod \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\" (UID: \"8d74fc7f-56d1-40ef-8041-f60ba05668a4\") " Jul 24 14:11:51.686458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686394 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") pod \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\" (UID: \"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d\") " Jul 24 14:11:51.686547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686436 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" (UID: "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:11:51.686547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686523 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b66cfea2-86c3-4f83-8bf6-f876eb10a7da" (UID: "b66cfea2-86c3-4f83-8bf6-f876eb10a7da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:11:51.686706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686687 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d74fc7f-56d1-40ef-8041-f60ba05668a4" (UID: "8d74fc7f-56d1-40ef-8041-f60ba05668a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:11:51.686782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686761 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.686838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686787 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.686838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.686816 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d74fc7f-56d1-40ef-8041-f60ba05668a4-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.688398 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.688366 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" (UID: "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:11:51.688398 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.688387 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664" (OuterVolumeSpecName: "kube-api-access-l8664") pod "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" (UID: "3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d"). InnerVolumeSpecName "kube-api-access-l8664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:11:51.688538 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.688410 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj" (OuterVolumeSpecName: "kube-api-access-6kjfj") pod "8d74fc7f-56d1-40ef-8041-f60ba05668a4" (UID: "8d74fc7f-56d1-40ef-8041-f60ba05668a4"). InnerVolumeSpecName "kube-api-access-6kjfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:11:51.688538 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.688432 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q" (OuterVolumeSpecName: "kube-api-access-7d58q") pod "b66cfea2-86c3-4f83-8bf6-f876eb10a7da" (UID: "b66cfea2-86c3-4f83-8bf6-f876eb10a7da"). InnerVolumeSpecName "kube-api-access-7d58q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:11:51.787765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.787670 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.787765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.787704 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8664\" (UniqueName: \"kubernetes.io/projected/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d-kube-api-access-l8664\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.787765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.787714 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d58q\" (UniqueName: \"kubernetes.io/projected/b66cfea2-86c3-4f83-8bf6-f876eb10a7da-kube-api-access-7d58q\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:51.787765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:51.787723 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjfj\" (UniqueName: \"kubernetes.io/projected/8d74fc7f-56d1-40ef-8041-f60ba05668a4-kube-api-access-6kjfj\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:11:52.179661 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.179625 2581 generic.go:356] "Generic (PLEG): container finished" podID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" containerID="cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7" exitCode=0 Jul 24 14:11:52.179819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.179680 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" Jul 24 14:11:52.179819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.179722 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" event={"ID":"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d","Type":"ContainerDied","Data":"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7"} Jul 24 14:11:52.179819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.179775 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7" event={"ID":"3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d","Type":"ContainerDied","Data":"60c980073add15a9fff269a69bdb680a6f7f9fd993a7b923539221ab6c1fa362"} Jul 24 14:11:52.179819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.179803 2581 scope.go:117] "RemoveContainer" containerID="cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7" Jul 24 14:11:52.181287 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.181256 2581 generic.go:356] "Generic (PLEG): container finished" podID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" containerID="b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194" exitCode=0 Jul 24 14:11:52.181383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.181300 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" Jul 24 14:11:52.181383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.181334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" event={"ID":"b66cfea2-86c3-4f83-8bf6-f876eb10a7da","Type":"ContainerDied","Data":"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194"} Jul 24 14:11:52.181383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.181381 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6" event={"ID":"b66cfea2-86c3-4f83-8bf6-f876eb10a7da","Type":"ContainerDied","Data":"5c1c447a251214dae0b3f9c3db9b9fb1416898a111dfca647fa21c2ace7ac11c"} Jul 24 14:11:52.182881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.182854 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" containerID="fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3" exitCode=0 Jul 24 14:11:52.183023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.182904 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" Jul 24 14:11:52.183023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.182920 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" event={"ID":"8d74fc7f-56d1-40ef-8041-f60ba05668a4","Type":"ContainerDied","Data":"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3"} Jul 24 14:11:52.183023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.182963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k" event={"ID":"8d74fc7f-56d1-40ef-8041-f60ba05668a4","Type":"ContainerDied","Data":"01971a6a76df5c7ef2ddfa92b8e84a7eb03d337570a726d262238b917e94f47e"} Jul 24 14:11:52.190620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.190596 2581 scope.go:117] "RemoveContainer" containerID="cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7" Jul 24 14:11:52.191056 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:11:52.190966 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7\": container with ID starting with cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7 not found: ID does not exist" containerID="cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7" Jul 24 14:11:52.191056 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.191001 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7"} err="failed to get container status \"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7\": rpc error: code = NotFound desc = could not find container \"cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7\": container with ID starting with cd7170509a405808bffe870e017c2461a4bf892a063f91ed0b49605dca8befd7 not found: ID does not exist" Jul 24 14:11:52.191056 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.191025 2581 scope.go:117] "RemoveContainer" containerID="b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194" Jul 24 14:11:52.200169 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.200151 2581 scope.go:117] "RemoveContainer" containerID="b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194" Jul 24 14:11:52.200478 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:11:52.200460 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194\": container with ID starting with b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194 not found: ID does not exist" containerID="b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194" Jul 24 14:11:52.200529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.200488 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194"} err="failed to get container status \"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194\": rpc error: code = NotFound desc = could not find container \"b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194\": container with ID starting with b52fb008b9876b178bbde5f95835ebeb110327753d083b2a2041eba15158d194 not found: ID does not exist" Jul 24 14:11:52.200529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.200507 2581 scope.go:117] "RemoveContainer" containerID="fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3" Jul 24 14:11:52.203231 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.203202 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:52.205740 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.205706 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-50cb7407-7d87d6f78f-7svt7"] Jul 24 14:11:52.209499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.209480 2581 scope.go:117] "RemoveContainer" containerID="fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3" Jul 24 14:11:52.209818 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:11:52.209797 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3\": container with ID starting with fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3 not found: ID does not exist" containerID="fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3" Jul 24 14:11:52.209879 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.209827 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3"} err="failed to get container status \"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3\": rpc error: code = NotFound desc = could not find container \"fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3\": container with ID starting with fc2bc6f70cee80f434727b58b6fd63cf48da219aace53845a1646c9e92571df3 not found: ID does not exist" Jul 24 14:11:52.213970 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.213944 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:52.216901 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.216877 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-50cb7407-76b6987c66-bzgw6"] Jul 24 14:11:52.225040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.225010 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:52.230119 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.230078 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-50cb7407-7bcc56f844-96j8k"] Jul 24 14:11:52.915850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.915814 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" path="/var/lib/kubelet/pods/3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d/volumes" Jul 24 14:11:52.916178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.916165 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" path="/var/lib/kubelet/pods/8d74fc7f-56d1-40ef-8041-f60ba05668a4/volumes" Jul 24 14:11:52.916467 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:52.916456 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" path="/var/lib/kubelet/pods/b66cfea2-86c3-4f83-8bf6-f876eb10a7da/volumes" Jul 24 14:11:55.997491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:55.997449 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk"] Jul 24 14:11:56.002286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:56.002258 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50cb7407-tnxxk"] Jul 24 14:11:56.004764 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:56.004738 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:56.594333 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:56.594300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:11:56.916009 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:11:56.915982 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5846e7-621b-40e8-ad7c-741d13587cc7" path="/var/lib/kubelet/pods/ba5846e7-621b-40e8-ad7c-741d13587cc7/volumes" Jul 24 14:12:02.664191 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:02.664132 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:02.771305 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:02.771272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:02.926121 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:02.926090 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:12:02.926647 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:02.926615 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" containerID="cri-o://d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813" gracePeriod=30 Jul 24 14:12:02.949650 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:02.949617 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:03.205240 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:03.205150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:03.311732 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:03.311698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:03.409259 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:03.409222 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:03.557671 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:03.557551 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:04.373276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:04.373240 2581 patch_prober.go:28] interesting pod/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 14:12:04.373788 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:04.373320 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 14:12:08.192755 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.192725 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:12:08.264000 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.263907 2581 generic.go:356] "Generic (PLEG): container finished" podID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerID="d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813" exitCode=0 Jul 24 14:12:08.264000 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.263963 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" Jul 24 14:12:08.264179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.264005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerDied","Data":"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813"} Jul 24 14:12:08.264179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.264067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc" event={"ID":"e4784e70-335b-4198-85a1-83fc4bc1dbf4","Type":"ContainerDied","Data":"d6fe3edddbd07d9ac342d80eece3ad68d0db5b9ca7c7961b88def7fbbe746dce"} Jul 24 14:12:08.264179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.264089 2581 scope.go:117] "RemoveContainer" containerID="d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813" Jul 24 14:12:08.272627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.272604 2581 scope.go:117] "RemoveContainer" containerID="d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e" Jul 24 14:12:08.280998 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.280975 2581 scope.go:117] "RemoveContainer" containerID="d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813" Jul 24 14:12:08.281326 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:12:08.281298 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813\": container with ID starting with d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813 not found: ID does not exist" containerID="d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813" Jul 24 14:12:08.281385 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.281336 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813"} err="failed to get container status \"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813\": rpc error: code = NotFound desc = could not find container \"d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813\": container with ID starting with d7d358a4848b64da57fbfa0907c11598849cb0e6aad78b4cb06933ff13b48813 not found: ID does not exist" Jul 24 14:12:08.281385 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.281355 2581 scope.go:117] "RemoveContainer" containerID="d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e" Jul 24 14:12:08.281652 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:12:08.281632 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e\": container with ID starting with d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e not found: ID does not exist" containerID="d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e" Jul 24 14:12:08.281719 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.281659 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e"} err="failed to get container status \"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e\": rpc error: code = NotFound desc = could not find container \"d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e\": container with ID starting with d3dd056dbc4ecdf682b5cba7420c41c006c649f2de2bbb4eafe1569189e2637e not found: ID does not exist" Jul 24 14:12:08.336008 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.335973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336008 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336013 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz58r\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336035 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336067 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336122 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336146 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336190 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336239 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336273 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket\") pod \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\" (UID: \"e4784e70-335b-4198-85a1-83fc4bc1dbf4\") " Jul 24 14:12:08.336669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336450 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:12:08.336669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336473 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:12:08.336669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336658 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.336796 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336679 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.336796 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336687 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:12:08.336982 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.336960 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:12:08.337324 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.337300 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:12:08.338420 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.338389 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r" (OuterVolumeSpecName: "kube-api-access-dz58r") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "kube-api-access-dz58r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:12:08.338528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.338506 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:12:08.338684 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.338667 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:12:08.338810 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.338793 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4784e70-335b-4198-85a1-83fc4bc1dbf4" (UID: "e4784e70-335b-4198-85a1-83fc4bc1dbf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:12:08.437930 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437889 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.437930 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437920 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.437930 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437931 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.438189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437943 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.438189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437955 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.438189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437968 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz58r\" (UniqueName: \"kubernetes.io/projected/e4784e70-335b-4198-85a1-83fc4bc1dbf4-kube-api-access-dz58r\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.438189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.437982 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4784e70-335b-4198-85a1-83fc4bc1dbf4-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:12:08.587276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.587238 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:12:08.592788 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.592753 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50cb7407-openshift-default-54554dd776-kn8bc"] Jul 24 14:12:08.916639 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:08.916607 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" path="/var/lib/kubelet/pods/e4784e70-335b-4198-85a1-83fc4bc1dbf4/volumes" Jul 24 14:12:09.597048 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:09.597016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:09.703772 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:09.703744 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:09.797819 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:09.797787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:09.968447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:09.968420 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:15.702935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:15.702898 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:16.213900 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:16.213862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:22.720738 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:22.720698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:23.013019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.012936 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:12:23.013297 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013285 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013298 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013310 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" containerName="payload-pre-processing" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013317 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" containerName="payload-pre-processing" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013325 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba5846e7-621b-40e8-ad7c-741d13587cc7" containerName="revoke-keys" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013331 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba5846e7-621b-40e8-ad7c-741d13587cc7" containerName="revoke-keys" Jul 24 14:12:23.013343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013341 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" containerName="payload-processing" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013347 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" containerName="payload-processing" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013356 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" containerName="maas-api" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013361 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" containerName="maas-api" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013371 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013376 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013434 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013442 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3825a6e4-85cc-4e53-b91a-bf29a5bb1d5d" containerName="maas-api" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013449 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66cfea2-86c3-4f83-8bf6-f876eb10a7da" containerName="payload-processing" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013455 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d74fc7f-56d1-40ef-8041-f60ba05668a4" containerName="payload-pre-processing" Jul 24 14:12:23.013524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.013463 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba5846e7-621b-40e8-ad7c-741d13587cc7" containerName="revoke-keys" Jul 24 14:12:23.018063 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.018037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.021294 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.021270 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-54be8ac5-openshift-default-dockercfg-66g6h\"" Jul 24 14:12:23.021980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.021961 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 14:12:23.032454 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.032415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:23.032959 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.032936 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:12:23.065752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48t6\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.065943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.065943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065811 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.065943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.065943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.066159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.066159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.065980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.066159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.066008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.066159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.066071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.166635 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.166578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.166977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.166948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167124 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.166995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167124 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167124 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167124 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167117 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48t6\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167208 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167576 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167537 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.167953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.167929 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.168106 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.168048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.168219 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.168180 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.170147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.170121 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.170423 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.170403 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.175828 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.175795 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.176016 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.175994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48t6\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6\") pod \"e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.203777 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.203736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:23.329207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.329109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:23.475676 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.475648 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:12:23.477717 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:12:23.477685 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dc5c7c_0fd4_4f56_acd8_a858a833e458.slice/crio-8a52abf40f9f72d0aec9f16040012195c1a2597dabbd47aa066827493f81561b WatchSource:0}: Error finding container 8a52abf40f9f72d0aec9f16040012195c1a2597dabbd47aa066827493f81561b: Status 404 returned error can't find the container with id 8a52abf40f9f72d0aec9f16040012195c1a2597dabbd47aa066827493f81561b Jul 24 14:12:23.480004 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.479946 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:23.480128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.480053 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:23.480128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:23.480098 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:24.343226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:24.343191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerStarted","Data":"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13"} Jul 24 14:12:24.343654 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:24.343281 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerStarted","Data":"8a52abf40f9f72d0aec9f16040012195c1a2597dabbd47aa066827493f81561b"} Jul 24 14:12:24.367397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:24.367336 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podStartSLOduration=2.367319698 podStartE2EDuration="2.367319698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:24.365276732 +0000 UTC m=+2037.961840806" watchObservedRunningTime="2026-07-24 14:12:24.367319698 +0000 UTC m=+2037.963883754" Jul 24 14:12:25.329696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:25.329653 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:25.335914 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:25.335878 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:25.348012 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:25.347984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:25.349120 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:25.349093 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:26.332398 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.332340 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:26.430050 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.430017 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:26.546926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.546889 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:12:26.547567 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.547551 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4784e70-335b-4198-85a1-83fc4bc1dbf4" containerName="istio-proxy" Jul 24 14:12:26.551055 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.551029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.553348 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.553325 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-54be8ac5-serving-cert\"" Jul 24 14:12:26.553493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.553396 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:12:26.560723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.560533 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:12:26.577984 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.577952 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:26.598266 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.598172 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgpx\" (UniqueName: \"kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.598266 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.598251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.598511 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.598279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.699887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.699847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.699887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.699890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.700168 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.699958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgpx\" (UniqueName: \"kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.700668 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.700645 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.702455 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.702433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.712147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.712117 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgpx\" (UniqueName: \"kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx\") pod \"maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:26.740044 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.740005 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:26.867001 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.866908 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:12:26.874885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:26.874858 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:27.033094 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.033070 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:12:27.035256 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:12:27.035208 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e91dd8_f731_47a5_a0ab_997079f2e3d7.slice/crio-90c2b553f73cc6712a210334942d594cec7a1b2c241f5e6e4e1f5f1a4f2b3e40 WatchSource:0}: Error finding container 90c2b553f73cc6712a210334942d594cec7a1b2c241f5e6e4e1f5f1a4f2b3e40: Status 404 returned error can't find the container with id 90c2b553f73cc6712a210334942d594cec7a1b2c241f5e6e4e1f5f1a4f2b3e40 Jul 24 14:12:27.036825 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.036807 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:12:27.103369 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.103335 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:27.215295 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.215264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:27.220768 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.220737 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:12:27.225745 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.225718 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.228164 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.228141 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-54be8ac5\"" Jul 24 14:12:27.237128 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.237096 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:12:27.253759 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.253726 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:12:27.257424 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.257404 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.259922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.259900 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-54be8ac5-dockercfg-5nlr8\"" Jul 24 14:12:27.267058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.267031 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:12:27.305870 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.305834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.306058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.305885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5zz\" (UniqueName: \"kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.306058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.305921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8f55\" (UniqueName: \"kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.306058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.305993 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.359835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.359796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" event={"ID":"78e91dd8-f731-47a5-a0ab-997079f2e3d7","Type":"ContainerStarted","Data":"90c2b553f73cc6712a210334942d594cec7a1b2c241f5e6e4e1f5f1a4f2b3e40"} Jul 24 14:12:27.407152 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.407056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8f55\" (UniqueName: \"kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.407152 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.407108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.407408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.407218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.407408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.407280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5zz\" (UniqueName: \"kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.407972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.407947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.408074 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.408027 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.415095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.415065 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5zz\" (UniqueName: \"kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz\") pod \"payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.415217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.415094 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8f55\" (UniqueName: \"kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55\") pod \"payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.539278 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.538043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:27.569973 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.569933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:27.699740 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.699699 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:12:27.703099 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:12:27.703038 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cac9e22_4683_4d06_b2eb_8451b9dab7c4.slice/crio-bb6567451cef2603314b37aa7838cf50b596ba4936a8efd471f5fe732dd7e629 WatchSource:0}: Error finding container bb6567451cef2603314b37aa7838cf50b596ba4936a8efd471f5fe732dd7e629: Status 404 returned error can't find the container with id bb6567451cef2603314b37aa7838cf50b596ba4936a8efd471f5fe732dd7e629 Jul 24 14:12:27.736815 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:27.736790 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:12:27.750002 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:12:27.749965 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40680966_5995_48b9_9630_84e16d49ef8b.slice/crio-0fb1b8803d79867463442b485918b4a8d1bc9a508915b19a27fbcce41bfdf056 WatchSource:0}: Error finding container 0fb1b8803d79867463442b485918b4a8d1bc9a508915b19a27fbcce41bfdf056: Status 404 returned error can't find the container with id 0fb1b8803d79867463442b485918b4a8d1bc9a508915b19a27fbcce41bfdf056 Jul 24 14:12:28.371108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.370973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" event={"ID":"40680966-5995-48b9-9630-84e16d49ef8b","Type":"ContainerStarted","Data":"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56"} Jul 24 14:12:28.371108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.371039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" event={"ID":"40680966-5995-48b9-9630-84e16d49ef8b","Type":"ContainerStarted","Data":"0fb1b8803d79867463442b485918b4a8d1bc9a508915b19a27fbcce41bfdf056"} Jul 24 14:12:28.371108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.371070 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:28.373281 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.373231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" event={"ID":"78e91dd8-f731-47a5-a0ab-997079f2e3d7","Type":"ContainerStarted","Data":"a7037630a13d7415e115fed35453e55541e7dffd991bcf685b7320a5d187056d"} Jul 24 14:12:28.373463 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.373376 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:28.375018 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.374991 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" event={"ID":"9cac9e22-4683-4d06-b2eb-8451b9dab7c4","Type":"ContainerStarted","Data":"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985"} Jul 24 14:12:28.375171 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.375039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" event={"ID":"9cac9e22-4683-4d06-b2eb-8451b9dab7c4","Type":"ContainerStarted","Data":"bb6567451cef2603314b37aa7838cf50b596ba4936a8efd471f5fe732dd7e629"} Jul 24 14:12:28.375171 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.375105 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:28.391806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.391751 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" podStartSLOduration=1.391733955 podStartE2EDuration="1.391733955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:28.39049072 +0000 UTC m=+2041.987054776" watchObservedRunningTime="2026-07-24 14:12:28.391733955 +0000 UTC m=+2041.988298009" Jul 24 14:12:28.409605 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.409519 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" podStartSLOduration=2.025061134 podStartE2EDuration="2.409503839s" totalImagesPullingTime="384.442705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:28.408054913 +0000 UTC m=+2042.004618968" watchObservedRunningTime="2026-07-24 14:12:28.409503839 +0000 UTC m=+2042.006067893" Jul 24 14:12:28.424658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:28.424602 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" podStartSLOduration=1.424562533 podStartE2EDuration="1.424562533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:28.423464618 +0000 UTC m=+2042.020028672" watchObservedRunningTime="2026-07-24 14:12:28.424562533 +0000 UTC m=+2042.021126587" Jul 24 14:12:30.839397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:30.839365 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:34.389180 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:34.389151 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:12:39.384778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:39.384739 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:12:39.385230 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:39.384959 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:12:41.143374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.143325 2581 patch_prober.go:28] interesting pod/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:59840->10.134.0.71:15021: read: connection reset by peer" start-of-body= Jul 24 14:12:41.143802 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.143402 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": read tcp 10.134.0.2:59840->10.134.0.71:15021: read: connection reset by peer" Jul 24 14:12:41.450881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.450849 2581 generic.go:356] "Generic (PLEG): container finished" podID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerID="7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13" exitCode=0 Jul 24 14:12:41.451071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.450924 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerDied","Data":"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13"} Jul 24 14:12:41.451391 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.451371 2581 scope.go:117] "RemoveContainer" containerID="7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13" Jul 24 14:12:41.452441 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.452411 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:41.452498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.452483 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:41.452537 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:41.452519 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:12:42.458197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:42.458163 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerStarted","Data":"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac"} Jul 24 14:12:43.330078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:43.330029 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:43.330303 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:43.330276 2581 patch_prober.go:28] interesting pod/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 24 14:12:43.330380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:43.330356 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 24 14:12:44.329735 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:44.329698 2581 patch_prober.go:28] interesting pod/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 24 14:12:44.330143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:44.329769 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 24 14:12:45.330305 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:45.330262 2581 patch_prober.go:28] interesting pod/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 24 14:12:45.330830 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:45.330342 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 24 14:12:46.330009 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:46.329972 2581 patch_prober.go:28] interesting pod/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Jul 24 14:12:46.330210 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:46.330052 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Jul 24 14:12:47.333727 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:47.333695 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:47.334194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:47.333987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:47.334782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:47.334764 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:12:54.234066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.233987 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:12:54.237554 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.237536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.241393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.241372 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-54be8ac5\"/\"default-dockercfg-hx44z\"" Jul 24 14:12:54.241530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.241413 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-54be8ac5\"/\"openshift-service-ca.crt\"" Jul 24 14:12:54.241530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.241438 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-54be8ac5\"/\"e2e-lifecycle-model-54be8ac5-kserve-self-signed-certs\"" Jul 24 14:12:54.241530 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.241458 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-54be8ac5\"/\"kube-root-ca.crt\"" Jul 24 14:12:54.254770 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.254737 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:12:54.261297 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.261249 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:12:54.363756 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.363972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363768 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.363972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363834 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.363972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.363972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4b7j\" (UniqueName: \"kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.363972 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.363932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465413 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465413 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4b7j\" (UniqueName: \"kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465693 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.465945 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465912 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.466007 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.465982 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.466111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.466093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.467776 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.467753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.468521 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.468501 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.473998 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.473974 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4b7j\" (UniqueName: \"kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j\") pod \"e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.551473 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.551381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:12:54.694960 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:54.694933 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:12:54.696595 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:12:54.696552 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6bee0a_2555_4243_8da0_5cfb35aebb62.slice/crio-9a1c6de728e8522f4f223bdf10c55b6881b939033e2a49c8d5ef3770dc1da1bf WatchSource:0}: Error finding container 9a1c6de728e8522f4f223bdf10c55b6881b939033e2a49c8d5ef3770dc1da1bf: Status 404 returned error can't find the container with id 9a1c6de728e8522f4f223bdf10c55b6881b939033e2a49c8d5ef3770dc1da1bf Jul 24 14:12:55.541690 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:55.541654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" event={"ID":"bd6bee0a-2555-4243-8da0-5cfb35aebb62","Type":"ContainerStarted","Data":"5ade2d4df2d7364ba2e772007d84e24946c4f59710fa6f0353c02efd4c265516"} Jul 24 14:12:55.542097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:55.541699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" event={"ID":"bd6bee0a-2555-4243-8da0-5cfb35aebb62","Type":"ContainerStarted","Data":"9a1c6de728e8522f4f223bdf10c55b6881b939033e2a49c8d5ef3770dc1da1bf"} Jul 24 14:12:55.559299 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:12:55.559249 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" podStartSLOduration=1.559229969 podStartE2EDuration="1.559229969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:55.558626592 +0000 UTC m=+2069.155190646" watchObservedRunningTime="2026-07-24 14:12:55.559229969 +0000 UTC m=+2069.155794026" Jul 24 14:13:04.551535 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:04.551491 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:13:04.552065 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:04.551640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:13:04.564739 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:04.564710 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:13:04.598230 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:04.598193 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:13:20.795935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:20.795894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:36.580633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.580568 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:13:36.581147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.580893 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" podUID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" containerName="maas-api" containerID="cri-o://a7037630a13d7415e115fed35453e55541e7dffd991bcf685b7320a5d187056d" gracePeriod=30 Jul 24 14:13:36.619308 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.619248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:36.698151 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.698114 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:13:36.698428 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.698401 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" podUID="40680966-5995-48b9-9630-84e16d49ef8b" containerName="payload-processing" containerID="cri-o://b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56" gracePeriod=30 Jul 24 14:13:36.730734 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.730699 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:13:36.730980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.730929 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" podUID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" containerName="payload-pre-processing" containerID="cri-o://f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985" gracePeriod=30 Jul 24 14:13:36.765800 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.765759 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:36.768174 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.768141 2581 generic.go:356] "Generic (PLEG): container finished" podID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" containerID="a7037630a13d7415e115fed35453e55541e7dffd991bcf685b7320a5d187056d" exitCode=0 Jul 24 14:13:36.768406 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.768386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" event={"ID":"78e91dd8-f731-47a5-a0ab-997079f2e3d7","Type":"ContainerDied","Data":"a7037630a13d7415e115fed35453e55541e7dffd991bcf685b7320a5d187056d"} Jul 24 14:13:36.872992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.872938 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:36.876432 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.876405 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:13:36.947148 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.947117 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls\") pod \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " Jul 24 14:13:36.947323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.947173 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsgpx\" (UniqueName: \"kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx\") pod \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " Jul 24 14:13:36.947323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.947255 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca\") pod \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\" (UID: \"78e91dd8-f731-47a5-a0ab-997079f2e3d7\") " Jul 24 14:13:36.947727 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.947693 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78e91dd8-f731-47a5-a0ab-997079f2e3d7" (UID: "78e91dd8-f731-47a5-a0ab-997079f2e3d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:13:36.949771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.949731 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx" (OuterVolumeSpecName: "kube-api-access-xsgpx") pod "78e91dd8-f731-47a5-a0ab-997079f2e3d7" (UID: "78e91dd8-f731-47a5-a0ab-997079f2e3d7"). InnerVolumeSpecName "kube-api-access-xsgpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:13:36.950129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:36.950046 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78e91dd8-f731-47a5-a0ab-997079f2e3d7" (UID: "78e91dd8-f731-47a5-a0ab-997079f2e3d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:13:37.032281 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.032255 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:13:37.036747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.036720 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:13:37.048058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.048023 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e91dd8-f731-47a5-a0ab-997079f2e3d7-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.048058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.048052 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsgpx\" (UniqueName: \"kubernetes.io/projected/78e91dd8-f731-47a5-a0ab-997079f2e3d7-kube-api-access-xsgpx\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.048058 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.048063 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e91dd8-f731-47a5-a0ab-997079f2e3d7-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.149417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149318 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8f55\" (UniqueName: \"kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55\") pod \"40680966-5995-48b9-9630-84e16d49ef8b\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " Jul 24 14:13:37.149417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149360 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj5zz\" (UniqueName: \"kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz\") pod \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " Jul 24 14:13:37.149417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149382 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume\") pod \"40680966-5995-48b9-9630-84e16d49ef8b\" (UID: \"40680966-5995-48b9-9630-84e16d49ef8b\") " Jul 24 14:13:37.149417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149411 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume\") pod \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\" (UID: \"9cac9e22-4683-4d06-b2eb-8451b9dab7c4\") " Jul 24 14:13:37.149837 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149808 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40680966-5995-48b9-9630-84e16d49ef8b" (UID: "40680966-5995-48b9-9630-84e16d49ef8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:13:37.149886 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.149866 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cac9e22-4683-4d06-b2eb-8451b9dab7c4" (UID: "9cac9e22-4683-4d06-b2eb-8451b9dab7c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:13:37.151688 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.151666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz" (OuterVolumeSpecName: "kube-api-access-pj5zz") pod "9cac9e22-4683-4d06-b2eb-8451b9dab7c4" (UID: "9cac9e22-4683-4d06-b2eb-8451b9dab7c4"). InnerVolumeSpecName "kube-api-access-pj5zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:13:37.151736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.151689 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55" (OuterVolumeSpecName: "kube-api-access-v8f55") pod "40680966-5995-48b9-9630-84e16d49ef8b" (UID: "40680966-5995-48b9-9630-84e16d49ef8b"). InnerVolumeSpecName "kube-api-access-v8f55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:13:37.250226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.250183 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.250226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.250218 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8f55\" (UniqueName: \"kubernetes.io/projected/40680966-5995-48b9-9630-84e16d49ef8b-kube-api-access-v8f55\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.250226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.250227 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj5zz\" (UniqueName: \"kubernetes.io/projected/9cac9e22-4683-4d06-b2eb-8451b9dab7c4-kube-api-access-pj5zz\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.250226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.250236 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40680966-5995-48b9-9630-84e16d49ef8b-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:13:37.775489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.775448 2581 generic.go:356] "Generic (PLEG): container finished" podID="40680966-5995-48b9-9630-84e16d49ef8b" containerID="b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56" exitCode=0 Jul 24 14:13:37.775950 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.775506 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" Jul 24 14:13:37.775950 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.775541 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" event={"ID":"40680966-5995-48b9-9630-84e16d49ef8b","Type":"ContainerDied","Data":"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56"} Jul 24 14:13:37.775950 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.775610 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974" event={"ID":"40680966-5995-48b9-9630-84e16d49ef8b","Type":"ContainerDied","Data":"0fb1b8803d79867463442b485918b4a8d1bc9a508915b19a27fbcce41bfdf056"} Jul 24 14:13:37.775950 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.775635 2581 scope.go:117] "RemoveContainer" containerID="b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56" Jul 24 14:13:37.777217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.777197 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" Jul 24 14:13:37.777396 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.777206 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm" event={"ID":"78e91dd8-f731-47a5-a0ab-997079f2e3d7","Type":"ContainerDied","Data":"90c2b553f73cc6712a210334942d594cec7a1b2c241f5e6e4e1f5f1a4f2b3e40"} Jul 24 14:13:37.778842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.778816 2581 generic.go:356] "Generic (PLEG): container finished" podID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" containerID="f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985" exitCode=0 Jul 24 14:13:37.779081 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.778853 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" event={"ID":"9cac9e22-4683-4d06-b2eb-8451b9dab7c4","Type":"ContainerDied","Data":"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985"} Jul 24 14:13:37.779081 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.778859 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" Jul 24 14:13:37.779081 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.778891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm" event={"ID":"9cac9e22-4683-4d06-b2eb-8451b9dab7c4","Type":"ContainerDied","Data":"bb6567451cef2603314b37aa7838cf50b596ba4936a8efd471f5fe732dd7e629"} Jul 24 14:13:37.786481 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.786459 2581 scope.go:117] "RemoveContainer" containerID="b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56" Jul 24 14:13:37.786916 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:13:37.786891 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56\": container with ID starting with b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56 not found: ID does not exist" containerID="b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56" Jul 24 14:13:37.787141 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.786928 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56"} err="failed to get container status \"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56\": rpc error: code = NotFound desc = could not find container \"b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56\": container with ID starting with b3ea179a8dda149c71a6fa929d66d67f55b701e1f2d05a10156344da3bea4d56 not found: ID does not exist" Jul 24 14:13:37.787141 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.786952 2581 scope.go:117] "RemoveContainer" containerID="a7037630a13d7415e115fed35453e55541e7dffd991bcf685b7320a5d187056d" Jul 24 14:13:37.798571 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.798549 2581 scope.go:117] "RemoveContainer" containerID="f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985" Jul 24 14:13:37.807082 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.807062 2581 scope.go:117] "RemoveContainer" containerID="f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985" Jul 24 14:13:37.807376 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:13:37.807359 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985\": container with ID starting with f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985 not found: ID does not exist" containerID="f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985" Jul 24 14:13:37.807444 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.807383 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985"} err="failed to get container status \"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985\": rpc error: code = NotFound desc = could not find container \"f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985\": container with ID starting with f78657c46504e1b7eb143c1d1d38381033786732f79ca954b21e9d2c4b9b8985 not found: ID does not exist" Jul 24 14:13:37.812651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.812621 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:13:37.822636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.822608 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54be8ac5-6f5877b6dd-qkwsm"] Jul 24 14:13:37.836724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.836691 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:13:37.845405 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.845326 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-54be8ac5-6bfbb8fd86-zx974"] Jul 24 14:13:37.867787 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.867753 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:13:37.873503 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:37.873471 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-54be8ac5-868b9d74cb-gbxvm"] Jul 24 14:13:38.916528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:38.916484 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40680966-5995-48b9-9630-84e16d49ef8b" path="/var/lib/kubelet/pods/40680966-5995-48b9-9630-84e16d49ef8b/volumes" Jul 24 14:13:38.917098 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:38.917077 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" path="/var/lib/kubelet/pods/78e91dd8-f731-47a5-a0ab-997079f2e3d7/volumes" Jul 24 14:13:38.917462 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:38.917444 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" path="/var/lib/kubelet/pods/9cac9e22-4683-4d06-b2eb-8451b9dab7c4/volumes" Jul 24 14:13:41.135749 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:41.135717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:44.732569 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:44.732533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:13:49.709786 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:49.709738 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:13:49.805961 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:13:49.805919 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:04.729332 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:04.729288 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" podUID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" containerName="main" containerID="cri-o://5ade2d4df2d7364ba2e772007d84e24946c4f59710fa6f0353c02efd4c265516" gracePeriod=45 Jul 24 14:14:04.917184 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:04.917146 2581 generic.go:356] "Generic (PLEG): container finished" podID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" containerID="5ade2d4df2d7364ba2e772007d84e24946c4f59710fa6f0353c02efd4c265516" exitCode=0 Jul 24 14:14:04.919401 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:04.919370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" event={"ID":"bd6bee0a-2555-4243-8da0-5cfb35aebb62","Type":"ContainerDied","Data":"5ade2d4df2d7364ba2e772007d84e24946c4f59710fa6f0353c02efd4c265516"} Jul 24 14:14:04.982612 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:04.982508 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:14:05.087003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.086972 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087022 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4b7j\" (UniqueName: \"kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087050 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087087 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087143 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087183 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir\") pod \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\" (UID: \"bd6bee0a-2555-4243-8da0-5cfb35aebb62\") " Jul 24 14:14:05.087461 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087347 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home" (OuterVolumeSpecName: "home") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:05.087461 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087455 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-home\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.087568 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087490 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:05.087568 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.087537 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:05.089366 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.089330 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm" (OuterVolumeSpecName: "dshm") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:05.089491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.089392 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j" (OuterVolumeSpecName: "kube-api-access-d4b7j") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "kube-api-access-d4b7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:14:05.089491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.089414 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd6bee0a-2555-4243-8da0-5cfb35aebb62" (UID: "bd6bee0a-2555-4243-8da0-5cfb35aebb62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:14:05.188677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.188636 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-dshm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.188677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.188669 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-model-cache\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.188677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.188681 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tmp-dir\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.188677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.188689 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6bee0a-2555-4243-8da0-5cfb35aebb62-tls-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.188995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.188699 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4b7j\" (UniqueName: \"kubernetes.io/projected/bd6bee0a-2555-4243-8da0-5cfb35aebb62-kube-api-access-d4b7j\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:05.923745 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.923717 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" Jul 24 14:14:05.924206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.923744 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw" event={"ID":"bd6bee0a-2555-4243-8da0-5cfb35aebb62","Type":"ContainerDied","Data":"9a1c6de728e8522f4f223bdf10c55b6881b939033e2a49c8d5ef3770dc1da1bf"} Jul 24 14:14:05.924206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.923798 2581 scope.go:117] "RemoveContainer" containerID="5ade2d4df2d7364ba2e772007d84e24946c4f59710fa6f0353c02efd4c265516" Jul 24 14:14:05.947521 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.947484 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:14:05.954827 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:05.954795 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-54be8ac5/e2e-lifecycle-model-54be8ac5-kserve-85b875bc59-dcpsw"] Jul 24 14:14:06.915778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:06.915745 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" path="/var/lib/kubelet/pods/bd6bee0a-2555-4243-8da0-5cfb35aebb62/volumes" Jul 24 14:14:27.779493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:27.779447 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:28.210727 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:28.210690 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:31.399184 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:31.399151 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:34.399270 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:34.399239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:38.551899 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:38.551864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:40.284979 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.284942 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:40.385289 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.385261 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:40.587708 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.587617 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:14:40.587870 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.587833 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" containerID="cri-o://d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac" gracePeriod=30 Jul 24 14:14:40.601047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.601015 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:40.843212 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:40.843125 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:42.030417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:42.030370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:42.136127 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:42.136093 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:42.258238 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:42.258196 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:43.166122 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:43.166084 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:43.261325 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:43.261292 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:43.349486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:43.349433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:44.966811 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:44.966775 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:45.845057 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.845028 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:14:45.941790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941753 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941799 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941821 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941843 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941880 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941931 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.941976 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942316 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942036 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k48t6\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942316 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942060 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket\") pod \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\" (UID: \"e6dc5c7c-0fd4-4f56-acd8-a858a833e458\") " Jul 24 14:14:45.942316 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942129 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:45.942316 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942279 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:45.942487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942358 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:45.942487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942356 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:45.942487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942374 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:14:45.942782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.942757 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data" (OuterVolumeSpecName: "istio-data") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:45.944205 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.944175 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:14:45.944337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.944294 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6" (OuterVolumeSpecName: "kube-api-access-k48t6") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "kube-api-access-k48t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:14:45.944337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.944293 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token" (OuterVolumeSpecName: "istio-token") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:14:45.944337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:45.944312 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e6dc5c7c-0fd4-4f56-acd8-a858a833e458" (UID: "e6dc5c7c-0fd4-4f56-acd8-a858a833e458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:14:46.043856 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043819 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.043856 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043849 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k48t6\" (UniqueName: \"kubernetes.io/projected/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-kube-api-access-k48t6\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.043856 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043859 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.044372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043867 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.044372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043878 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.044372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043885 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.044372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043892 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.044372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.043900 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6dc5c7c-0fd4-4f56-acd8-a858a833e458-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:14:46.129388 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.129347 2581 generic.go:356] "Generic (PLEG): container finished" podID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerID="d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac" exitCode=0 Jul 24 14:14:46.129540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.129421 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" Jul 24 14:14:46.129540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.129432 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerDied","Data":"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac"} Jul 24 14:14:46.129540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.129517 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2" event={"ID":"e6dc5c7c-0fd4-4f56-acd8-a858a833e458","Type":"ContainerDied","Data":"8a52abf40f9f72d0aec9f16040012195c1a2597dabbd47aa066827493f81561b"} Jul 24 14:14:46.129706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.129541 2581 scope.go:117] "RemoveContainer" containerID="d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac" Jul 24 14:14:46.139069 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.139050 2581 scope.go:117] "RemoveContainer" containerID="7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13" Jul 24 14:14:46.148223 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.148197 2581 scope.go:117] "RemoveContainer" containerID="d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac" Jul 24 14:14:46.148636 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:14:46.148575 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac\": container with ID starting with d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac not found: ID does not exist" containerID="d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac" Jul 24 14:14:46.148736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.148648 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac"} err="failed to get container status \"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac\": rpc error: code = NotFound desc = could not find container \"d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac\": container with ID starting with d2153187db9bd809a125ce0fd7e5b18403a720645623f405ff80deac76c964ac not found: ID does not exist" Jul 24 14:14:46.148736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.148668 2581 scope.go:117] "RemoveContainer" containerID="7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13" Jul 24 14:14:46.148943 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:14:46.148925 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13\": container with ID starting with 7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13 not found: ID does not exist" containerID="7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13" Jul 24 14:14:46.148986 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.148949 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13"} err="failed to get container status \"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13\": rpc error: code = NotFound desc = could not find container \"7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13\": container with ID starting with 7bbb957867c7426fde4f8d985c1004b281faad464978810b530b39bf7f2a7b13 not found: ID does not exist" Jul 24 14:14:46.153652 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.153620 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:14:46.158719 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.158691 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-54be8ac5-openshift-default-d9b95ccdd-2clt2"] Jul 24 14:14:46.916158 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:46.916117 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" path="/var/lib/kubelet/pods/e6dc5c7c-0fd4-4f56-acd8-a858a833e458/volumes" Jul 24 14:14:51.145118 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:51.145077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:51.283900 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:51.283863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:51.838719 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:51.838685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:58.237722 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:58.237685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:58.348345 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:58.348311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:14:58.443365 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:14:58.443335 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:00.148781 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.148747 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858"] Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149096 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149107 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149119 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" containerName="payload-pre-processing" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149124 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" containerName="payload-pre-processing" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149136 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" containerName="maas-api" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149141 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" containerName="maas-api" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149150 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" containerName="main" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149156 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" containerName="main" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149164 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40680966-5995-48b9-9630-84e16d49ef8b" containerName="payload-processing" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149170 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40680966-5995-48b9-9630-84e16d49ef8b" containerName="payload-processing" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149177 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149182 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149239 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149245 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6dc5c7c-0fd4-4f56-acd8-a858a833e458" containerName="istio-proxy" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149254 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e91dd8-f731-47a5-a0ab-997079f2e3d7" containerName="maas-api" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149260 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cac9e22-4683-4d06-b2eb-8451b9dab7c4" containerName="payload-pre-processing" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149265 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd6bee0a-2555-4243-8da0-5cfb35aebb62" containerName="main" Jul 24 14:15:00.149663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.149271 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40680966-5995-48b9-9630-84e16d49ef8b" containerName="payload-processing" Jul 24 14:15:00.154005 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.153982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" Jul 24 14:15:00.156667 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.156640 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:15:00.159922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.159889 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858"] Jul 24 14:15:00.167706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.167680 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" Jul 24 14:15:00.313490 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:00.313461 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858"] Jul 24 14:15:00.315188 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:00.315162 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac49f24_b184_4f72_b5ad_aa4bf12a4d46.slice/crio-d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6 WatchSource:0}: Error finding container d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6: Status 404 returned error can't find the container with id d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6 Jul 24 14:15:01.203547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:01.203503 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" event={"ID":"8ac49f24-b184-4f72-b5ad-aa4bf12a4d46","Type":"ContainerStarted","Data":"9f58994ab662a8bb3a20eba067b8b44542245b1b959ab49ab9380ee5cb82a179"} Jul 24 14:15:01.204032 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:01.203562 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" event={"ID":"8ac49f24-b184-4f72-b5ad-aa4bf12a4d46","Type":"ContainerStarted","Data":"d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6"} Jul 24 14:15:01.220604 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:01.220523 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" podStartSLOduration=1.220505547 podStartE2EDuration="1.220505547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:01.21891736 +0000 UTC m=+2194.815481415" watchObservedRunningTime="2026-07-24 14:15:01.220505547 +0000 UTC m=+2194.817069601" Jul 24 14:15:02.210150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:02.210119 2581 generic.go:356] "Generic (PLEG): container finished" podID="8ac49f24-b184-4f72-b5ad-aa4bf12a4d46" containerID="9f58994ab662a8bb3a20eba067b8b44542245b1b959ab49ab9380ee5cb82a179" exitCode=0 Jul 24 14:15:02.210667 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:02.210190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" event={"ID":"8ac49f24-b184-4f72-b5ad-aa4bf12a4d46","Type":"ContainerDied","Data":"9f58994ab662a8bb3a20eba067b8b44542245b1b959ab49ab9380ee5cb82a179"} Jul 24 14:15:03.355062 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:03.355028 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" Jul 24 14:15:04.222992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:04.222959 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" event={"ID":"8ac49f24-b184-4f72-b5ad-aa4bf12a4d46","Type":"ContainerDied","Data":"d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6"} Jul 24 14:15:04.222992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:04.222992 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-pb858" Jul 24 14:15:04.223216 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:04.223011 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64a903610172ab1f59ea7cb69df32e024b7b58287ef13a4f8e3bb8716e43ef6" Jul 24 14:15:09.013547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:09.013513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:09.419232 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:09.419203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:16.927800 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:16.927764 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:16.988900 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:16.988872 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:27.163163 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:27.163131 2581 scope.go:117] "RemoveContainer" containerID="92461aed939abb7b5e3ed83ec01301d3ec21e61207e808ab2b4daa82708f3816" Jul 24 14:15:30.445504 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:30.445459 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:36.984117 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:36.984078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:37.084179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.084144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:37.383392 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.383312 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:37.709156 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.709118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:15:37.709503 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.709487 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac49f24-b184-4f72-b5ad-aa4bf12a4d46" containerName="cleanup" Jul 24 14:15:37.709570 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.709503 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac49f24-b184-4f72-b5ad-aa4bf12a4d46" containerName="cleanup" Jul 24 14:15:37.709658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.709598 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac49f24-b184-4f72-b5ad-aa4bf12a4d46" containerName="cleanup" Jul 24 14:15:37.712380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.712358 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.715354 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.715328 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6304ef-openshift-default-dockercfg-kgd9j\"" Jul 24 14:15:37.715518 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.715443 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 14:15:37.732189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.732150 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:15:37.741504 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.741472 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:37.756417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756521 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756540 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756654 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxl9\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.756985 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.756805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxl9\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857438 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857767 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.857922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.857836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.858090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.858059 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.858478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.858451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.859798 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.859765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.859993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.859976 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.866914 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.866851 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:37.867193 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:37.867167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxl9\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9\") pod \"e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:38.023508 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.023405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:38.183732 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.183707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:15:38.185676 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:38.185644 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214e4513_a62d_4a6e_bb05_fd7baec0318c.slice/crio-8be47a0512021da84be4e1d20201eb8287c85389cfb83ad86683c566bba1f30a WatchSource:0}: Error finding container 8be47a0512021da84be4e1d20201eb8287c85389cfb83ad86683c566bba1f30a: Status 404 returned error can't find the container with id 8be47a0512021da84be4e1d20201eb8287c85389cfb83ad86683c566bba1f30a Jul 24 14:15:38.187906 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.187877 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:38.187985 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.187941 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:38.187985 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.187970 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:38.203024 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.202989 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:38.390560 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.390470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerStarted","Data":"ea035b53d91e5986a74d297de6dcc4978638e9228a41429477be32342e4002df"} Jul 24 14:15:38.390560 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.390530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerStarted","Data":"8be47a0512021da84be4e1d20201eb8287c85389cfb83ad86683c566bba1f30a"} Jul 24 14:15:38.414040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:38.413992 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podStartSLOduration=1.413977048 podStartE2EDuration="1.413977048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:38.412875083 +0000 UTC m=+2232.009439139" watchObservedRunningTime="2026-07-24 14:15:38.413977048 +0000 UTC m=+2232.010541104" Jul 24 14:15:39.024436 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:39.024396 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:39.029421 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:39.029391 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:39.396224 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:39.396119 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:39.397241 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:39.397222 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:40.431863 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.431819 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:40.529953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.529922 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:40.639680 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.639631 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:15:40.643673 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.643642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.647676 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.647648 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6304ef-serving-cert\"" Jul 24 14:15:40.647867 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.647705 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:15:40.657468 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.657388 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:15:40.683778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.683670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.683778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.683722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rwr\" (UniqueName: \"kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.683778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.683751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.684142 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.684092 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:40.784702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.784659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.784702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.784704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwr\" (UniqueName: \"kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.784986 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.784730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.784986 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:15:40.784831 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-serving-cert: secret "maas-api-e2e-inf-a-6304ef-serving-cert" not found Jul 24 14:15:40.784986 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:15:40.784919 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls podName:62a9ea99-a507-4766-bbc5-480533689bdf nodeName:}" failed. No retries permitted until 2026-07-24 14:15:41.284896486 +0000 UTC m=+2234.881460523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls") pod "maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" (UID: "62a9ea99-a507-4766-bbc5-480533689bdf") : secret "maas-api-e2e-inf-a-6304ef-serving-cert" not found Jul 24 14:15:40.785438 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.785417 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:40.795221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:40.795185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwr\" (UniqueName: \"kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:41.124542 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.124431 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:41.229984 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.229938 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:41.290528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.290491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:41.293073 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.293044 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:41.305034 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.304997 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:15:41.308376 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.308346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:41.308962 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.308943 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.311283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.311261 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6304ef\"" Jul 24 14:15:41.311419 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.311364 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6304ef-dockercfg-cmcjm\"" Jul 24 14:15:41.319386 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.319357 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:15:41.347961 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.347929 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:15:41.351269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.351249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.361425 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.361398 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:15:41.391349 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.391247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.391503 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.391375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7ch\" (UniqueName: \"kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.492535 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.492492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.493052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.492565 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.493052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.492703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsb55\" (UniqueName: \"kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.493052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.492753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7ch\" (UniqueName: \"kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.493346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.493326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.501291 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.501251 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7ch\" (UniqueName: \"kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch\") pod \"payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.558171 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.558132 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:41.593561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.593524 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.593759 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.593579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsb55\" (UniqueName: \"kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.594328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.594302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.602219 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.602182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsb55\" (UniqueName: \"kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55\") pod \"payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.621802 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.621759 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:41.667161 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.666804 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:41.725533 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.725496 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:15:41.729486 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:41.729434 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a9ea99_a507_4766_bbc5_480533689bdf.slice/crio-e1c5c303a332acc041c3a6a6d2a05ebd2dcd5a7edc0401a5edd292713a23b28c WatchSource:0}: Error finding container e1c5c303a332acc041c3a6a6d2a05ebd2dcd5a7edc0401a5edd292713a23b28c: Status 404 returned error can't find the container with id e1c5c303a332acc041c3a6a6d2a05ebd2dcd5a7edc0401a5edd292713a23b28c Jul 24 14:15:41.797164 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.797101 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:15:41.799670 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:41.799621 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744054df_ca54_4757_af37_28d3031db1b8.slice/crio-7e608cf3fad28684170c1c7005ea1a075cdd4e63038a6ca4c29a669e762d17ff WatchSource:0}: Error finding container 7e608cf3fad28684170c1c7005ea1a075cdd4e63038a6ca4c29a669e762d17ff: Status 404 returned error can't find the container with id 7e608cf3fad28684170c1c7005ea1a075cdd4e63038a6ca4c29a669e762d17ff Jul 24 14:15:41.849408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:41.849376 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:15:41.851613 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:41.851570 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde63b3e7_d07d_4498_8ee3_48490d6cb4db.slice/crio-4f917f4e001c35cdde67cc40f1065c7635e65e7f6d209f10dd745c08f7aaa074 WatchSource:0}: Error finding container 4f917f4e001c35cdde67cc40f1065c7635e65e7f6d209f10dd745c08f7aaa074: Status 404 returned error can't find the container with id 4f917f4e001c35cdde67cc40f1065c7635e65e7f6d209f10dd745c08f7aaa074 Jul 24 14:15:42.414378 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.414342 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" event={"ID":"de63b3e7-d07d-4498-8ee3-48490d6cb4db","Type":"ContainerStarted","Data":"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1"} Jul 24 14:15:42.414566 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.414411 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:42.414566 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.414435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" event={"ID":"de63b3e7-d07d-4498-8ee3-48490d6cb4db","Type":"ContainerStarted","Data":"4f917f4e001c35cdde67cc40f1065c7635e65e7f6d209f10dd745c08f7aaa074"} Jul 24 14:15:42.416070 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.416044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" event={"ID":"744054df-ca54-4757-af37-28d3031db1b8","Type":"ContainerStarted","Data":"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5"} Jul 24 14:15:42.416204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.416090 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" event={"ID":"744054df-ca54-4757-af37-28d3031db1b8","Type":"ContainerStarted","Data":"7e608cf3fad28684170c1c7005ea1a075cdd4e63038a6ca4c29a669e762d17ff"} Jul 24 14:15:42.416204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.416122 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:42.417522 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.417499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" event={"ID":"62a9ea99-a507-4766-bbc5-480533689bdf","Type":"ContainerStarted","Data":"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1"} Jul 24 14:15:42.417633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.417539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" event={"ID":"62a9ea99-a507-4766-bbc5-480533689bdf","Type":"ContainerStarted","Data":"e1c5c303a332acc041c3a6a6d2a05ebd2dcd5a7edc0401a5edd292713a23b28c"} Jul 24 14:15:42.417633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.417610 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:42.433347 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.433295 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" podStartSLOduration=1.433278613 podStartE2EDuration="1.433278613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:42.431565178 +0000 UTC m=+2236.028129233" watchObservedRunningTime="2026-07-24 14:15:42.433278613 +0000 UTC m=+2236.029842668" Jul 24 14:15:42.450745 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.450686 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" podStartSLOduration=1.450673429 podStartE2EDuration="1.450673429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:42.449188543 +0000 UTC m=+2236.045752597" watchObservedRunningTime="2026-07-24 14:15:42.450673429 +0000 UTC m=+2236.047237502" Jul 24 14:15:42.469026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:42.468898 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" podStartSLOduration=1.989370109 podStartE2EDuration="2.468872921s" totalImagesPullingTime="479.502812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:42.467425241 +0000 UTC m=+2236.063989302" watchObservedRunningTime="2026-07-24 14:15:42.468872921 +0000 UTC m=+2236.065436981" Jul 24 14:15:44.996437 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:44.996403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:45.378045 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.377957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:45.657332 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.657234 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:15:45.661168 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.661138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.664416 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.664386 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-406bb9-openshift-default-dockercfg-ql2jl\"" Jul 24 14:15:45.672207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.672172 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:15:45.685658 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.685571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:45.732876 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.732830 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.732912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.732962 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.732986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.733009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.733030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733079 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.733063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.733086 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.733300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.733105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834155 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834232 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.834774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834704 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.835015 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834972 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.835015 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.834973 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.835123 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.835024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.835211 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.835185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.836925 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.836893 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.837167 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.837148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.843156 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.843120 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.843304 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.843219 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g\") pod \"e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:45.975493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:45.975442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:46.031489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.031435 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:46.133480 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.133450 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:15:46.135496 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:46.135453 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf127ffe8_5110_445c_9d5c_7240e03306d2.slice/crio-613e7f9761e0fae7ba078f2e38013e9086e46dd7a72a372d554396277af9729c WatchSource:0}: Error finding container 613e7f9761e0fae7ba078f2e38013e9086e46dd7a72a372d554396277af9729c: Status 404 returned error can't find the container with id 613e7f9761e0fae7ba078f2e38013e9086e46dd7a72a372d554396277af9729c Jul 24 14:15:46.138178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.138139 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:46.138322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.138230 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:46.138322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.138284 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:46.439244 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.439203 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerStarted","Data":"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c"} Jul 24 14:15:46.439415 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.439266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerStarted","Data":"613e7f9761e0fae7ba078f2e38013e9086e46dd7a72a372d554396277af9729c"} Jul 24 14:15:46.461527 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.461471 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podStartSLOduration=1.461453557 podStartE2EDuration="1.461453557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:46.460511303 +0000 UTC m=+2240.057075357" watchObservedRunningTime="2026-07-24 14:15:46.461453557 +0000 UTC m=+2240.058017611" Jul 24 14:15:46.976041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:46.976006 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:47.981439 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:47.981401 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:48.429335 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.429307 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:15:48.450439 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.450409 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:48.451877 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.451851 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:15:48.550736 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.550697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:48.616686 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.616653 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:15:48.619896 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.619876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.622239 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.622213 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-406bb9-serving-cert\"" Jul 24 14:15:48.634194 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.634161 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:15:48.690753 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.690666 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:48.760677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.760638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6nw\" (UniqueName: \"kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.760902 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.760708 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.760902 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.760792 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.789220 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.789182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:48.862299 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.862260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6nw\" (UniqueName: \"kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.862509 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.862310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.862509 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.862430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.863158 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.863127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.865385 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.865348 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.874299 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.874270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6nw\" (UniqueName: \"kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw\") pod \"maas-api-e2e-inf-b-406bb9-774cc45d95-7874d\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:48.888519 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.888481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:48.935068 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:48.935029 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:49.091718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.091689 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:15:49.094300 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:49.094259 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54ed840_a4c5_4721_82e2_5c209c970183.slice/crio-3f63ba4c21196635092419397e595c0c02daf03e5532f8c6e1992ad25088ef44 WatchSource:0}: Error finding container 3f63ba4c21196635092419397e595c0c02daf03e5532f8c6e1992ad25088ef44: Status 404 returned error can't find the container with id 3f63ba4c21196635092419397e595c0c02daf03e5532f8c6e1992ad25088ef44 Jul 24 14:15:49.296450 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.296353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:49.344802 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.344769 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:15:49.349726 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.349695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.352689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.352665 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-406bb9-dockercfg-hwb4z\"" Jul 24 14:15:49.352993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.352975 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-406bb9\"" Jul 24 14:15:49.359450 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.359414 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:15:49.379778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.379748 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:49.457392 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.457357 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" event={"ID":"c54ed840-a4c5-4721-82e2-5c209c970183","Type":"ContainerStarted","Data":"3f63ba4c21196635092419397e595c0c02daf03e5532f8c6e1992ad25088ef44"} Jul 24 14:15:49.478155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.478127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.478277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.478248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47snm\" (UniqueName: \"kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.579539 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.579432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47snm\" (UniqueName: \"kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.579767 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.579558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.580281 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.580259 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.588535 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.588507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47snm\" (UniqueName: \"kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm\") pod \"payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.663678 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.663637 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:49.821790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:49.821763 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:15:49.824153 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:15:49.824122 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fec5c3d_f4dd_436c_be73_91a44db6ed51.slice/crio-a28ff69054fa17f41ef003c102a61a515b1c6ccee6f2884cb495d8708bef2e85 WatchSource:0}: Error finding container a28ff69054fa17f41ef003c102a61a515b1c6ccee6f2884cb495d8708bef2e85: Status 404 returned error can't find the container with id a28ff69054fa17f41ef003c102a61a515b1c6ccee6f2884cb495d8708bef2e85 Jul 24 14:15:50.466922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.465942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" event={"ID":"3fec5c3d-f4dd-436c-be73-91a44db6ed51","Type":"ContainerStarted","Data":"9c59234eba6c4e55f2302d3b026910ab41b1d9f833ca1a2cc9065d3b6cf02cdc"} Jul 24 14:15:50.466922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.466012 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" event={"ID":"3fec5c3d-f4dd-436c-be73-91a44db6ed51","Type":"ContainerStarted","Data":"a28ff69054fa17f41ef003c102a61a515b1c6ccee6f2884cb495d8708bef2e85"} Jul 24 14:15:50.466922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.466860 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:15:50.471630 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.471577 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" event={"ID":"c54ed840-a4c5-4721-82e2-5c209c970183","Type":"ContainerStarted","Data":"fd563c726d4324517ef6ac942c791c7e95e063adccb171b06ed5276b8914c950"} Jul 24 14:15:50.471966 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.471904 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:50.505568 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.505502 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" podStartSLOduration=1.505477155 podStartE2EDuration="1.505477155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:50.485786262 +0000 UTC m=+2244.082350318" watchObservedRunningTime="2026-07-24 14:15:50.505477155 +0000 UTC m=+2244.102041210" Jul 24 14:15:50.507049 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:50.506990 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" podStartSLOduration=2.127543169 podStartE2EDuration="2.506972064s" totalImagesPullingTime="379.428895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:50.504196016 +0000 UTC m=+2244.100760073" watchObservedRunningTime="2026-07-24 14:15:50.506972064 +0000 UTC m=+2244.103536119" Jul 24 14:15:53.069657 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:53.069620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:15:53.424400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:53.424366 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:15:53.424636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:53.424458 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:15:55.437807 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:55.437758 2581 patch_prober.go:28] interesting pod/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:15:55.438367 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:55.437835 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:15:56.485957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:56.485923 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:15:57.524264 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.524233 2581 generic.go:356] "Generic (PLEG): container finished" podID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerID="ea035b53d91e5986a74d297de6dcc4978638e9228a41429477be32342e4002df" exitCode=0 Jul 24 14:15:57.524769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.524312 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerDied","Data":"ea035b53d91e5986a74d297de6dcc4978638e9228a41429477be32342e4002df"} Jul 24 14:15:57.524769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.524736 2581 scope.go:117] "RemoveContainer" containerID="ea035b53d91e5986a74d297de6dcc4978638e9228a41429477be32342e4002df" Jul 24 14:15:57.525743 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.525716 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:57.525813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.525788 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:57.525854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:57.525819 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:15:58.532472 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:58.532435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerStarted","Data":"d6f0f44d09432a6d039f3ed5906c54bc8fdc3bcb4b9e9ca996f5982d0f7c644a"} Jul 24 14:15:59.024732 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:59.024689 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:15:59.024929 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:59.024861 2581 patch_prober.go:28] interesting pod/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 24 14:15:59.024974 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:15:59.024922 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 24 14:16:00.024507 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:00.024459 2581 patch_prober.go:28] interesting pod/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 24 14:16:00.024957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:00.024516 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 24 14:16:01.024197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:01.024160 2581 patch_prober.go:28] interesting pod/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 24 14:16:01.024388 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:01.024217 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 24 14:16:02.024825 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:02.024783 2581 patch_prober.go:28] interesting pod/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Jul 24 14:16:02.025310 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:02.024851 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Jul 24 14:16:02.490065 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:02.490023 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:16:03.027840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:03.027812 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:16:03.028310 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:03.028159 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:16:03.029003 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:03.028983 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:16:04.491811 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:04.491765 2581 patch_prober.go:28] interesting pod/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:16:04.492227 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:04.491840 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:16:06.579696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.579577 2581 generic.go:356] "Generic (PLEG): container finished" podID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerID="f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c" exitCode=0 Jul 24 14:16:06.579696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.579651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerDied","Data":"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c"} Jul 24 14:16:06.580198 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.580119 2581 scope.go:117] "RemoveContainer" containerID="f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c" Jul 24 14:16:06.581393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.581360 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:16:06.581528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.581435 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:16:06.581528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:06.581469 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:16:07.591669 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:07.591629 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerStarted","Data":"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84"} Jul 24 14:16:07.976134 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:07.976096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:16:07.976465 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:07.976169 2581 patch_prober.go:28] interesting pod/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Jul 24 14:16:07.976691 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:07.976658 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Jul 24 14:16:08.977032 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:08.976992 2581 patch_prober.go:28] interesting pod/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Jul 24 14:16:08.978169 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:08.977691 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Jul 24 14:16:09.976749 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:09.976711 2581 patch_prober.go:28] interesting pod/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Jul 24 14:16:09.977125 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:09.977083 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Jul 24 14:16:10.371544 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:10.371424 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:16:10.976462 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:10.975963 2581 patch_prober.go:28] interesting pod/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Jul 24 14:16:10.976462 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:10.976048 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Jul 24 14:16:11.980318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:11.980279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:16:11.980743 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:11.980556 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:16:11.981850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:11.981831 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:16:27.186574 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:27.186535 2581 scope.go:117] "RemoveContainer" containerID="f2bf74170c4e00df8f07b7d5f9195909d5a057509651ab8d2cb8b5277b8fa704" Jul 24 14:16:33.962308 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:16:33.962272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:13.004403 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:13.004372 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:13.950209 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:13.950168 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:24.513929 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.513831 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:17:24.514502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.514098 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" podUID="62a9ea99-a507-4766-bbc5-480533689bdf" containerName="maas-api" containerID="cri-o://358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1" gracePeriod=30 Jul 24 14:17:24.547611 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.547567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:24.576919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.576879 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:17:24.577191 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.577158 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" podUID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" containerName="payload-processing" containerID="cri-o://d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1" gracePeriod=30 Jul 24 14:17:24.585273 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.585236 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:17:24.585502 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.585475 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" podUID="744054df-ca54-4757-af37-28d3031db1b8" containerName="payload-pre-processing" containerID="cri-o://0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5" gracePeriod=30 Jul 24 14:17:24.874323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.874294 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:17:24.901947 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.901893 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:24.907240 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.907214 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:17:24.913612 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.913575 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:17:24.957955 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.957921 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") pod \"62a9ea99-a507-4766-bbc5-480533689bdf\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " Jul 24 14:17:24.958161 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.958068 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca\") pod \"62a9ea99-a507-4766-bbc5-480533689bdf\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " Jul 24 14:17:24.958161 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.958104 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56rwr\" (UniqueName: \"kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr\") pod \"62a9ea99-a507-4766-bbc5-480533689bdf\" (UID: \"62a9ea99-a507-4766-bbc5-480533689bdf\") " Jul 24 14:17:24.958875 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.958842 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62a9ea99-a507-4766-bbc5-480533689bdf" (UID: "62a9ea99-a507-4766-bbc5-480533689bdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:24.960742 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.960708 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62a9ea99-a507-4766-bbc5-480533689bdf" (UID: "62a9ea99-a507-4766-bbc5-480533689bdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:17:24.960742 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.960724 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr" (OuterVolumeSpecName: "kube-api-access-56rwr") pod "62a9ea99-a507-4766-bbc5-480533689bdf" (UID: "62a9ea99-a507-4766-bbc5-480533689bdf"). InnerVolumeSpecName "kube-api-access-56rwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:24.996374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:24.996339 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:25.008135 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.008102 2581 generic.go:356] "Generic (PLEG): container finished" podID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" containerID="d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1" exitCode=0 Jul 24 14:17:25.008355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.008153 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" Jul 24 14:17:25.008355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.008174 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" event={"ID":"de63b3e7-d07d-4498-8ee3-48490d6cb4db","Type":"ContainerDied","Data":"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1"} Jul 24 14:17:25.008355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.008231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq" event={"ID":"de63b3e7-d07d-4498-8ee3-48490d6cb4db","Type":"ContainerDied","Data":"4f917f4e001c35cdde67cc40f1065c7635e65e7f6d209f10dd745c08f7aaa074"} Jul 24 14:17:25.008355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.008267 2581 scope.go:117] "RemoveContainer" containerID="d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1" Jul 24 14:17:25.009523 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.009501 2581 generic.go:356] "Generic (PLEG): container finished" podID="744054df-ca54-4757-af37-28d3031db1b8" containerID="0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5" exitCode=0 Jul 24 14:17:25.009648 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.009547 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" Jul 24 14:17:25.009716 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.009615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" event={"ID":"744054df-ca54-4757-af37-28d3031db1b8","Type":"ContainerDied","Data":"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5"} Jul 24 14:17:25.009788 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.009758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q" event={"ID":"744054df-ca54-4757-af37-28d3031db1b8","Type":"ContainerDied","Data":"7e608cf3fad28684170c1c7005ea1a075cdd4e63038a6ca4c29a669e762d17ff"} Jul 24 14:17:25.010918 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.010894 2581 generic.go:356] "Generic (PLEG): container finished" podID="62a9ea99-a507-4766-bbc5-480533689bdf" containerID="358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1" exitCode=0 Jul 24 14:17:25.011041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.010937 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" Jul 24 14:17:25.011041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.010968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" event={"ID":"62a9ea99-a507-4766-bbc5-480533689bdf","Type":"ContainerDied","Data":"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1"} Jul 24 14:17:25.011041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.010998 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25" event={"ID":"62a9ea99-a507-4766-bbc5-480533689bdf","Type":"ContainerDied","Data":"e1c5c303a332acc041c3a6a6d2a05ebd2dcd5a7edc0401a5edd292713a23b28c"} Jul 24 14:17:25.017692 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.017674 2581 scope.go:117] "RemoveContainer" containerID="d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1" Jul 24 14:17:25.018015 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:17:25.017987 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1\": container with ID starting with d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1 not found: ID does not exist" containerID="d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1" Jul 24 14:17:25.018107 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.018028 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1"} err="failed to get container status \"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1\": rpc error: code = NotFound desc = could not find container \"d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1\": container with ID starting with d80f92f76ebbfac4c33dd057295229ac997f583bce178b411b6ca6f538e12ee1 not found: ID does not exist" Jul 24 14:17:25.018107 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.018047 2581 scope.go:117] "RemoveContainer" containerID="0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5" Jul 24 14:17:25.026989 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.026968 2581 scope.go:117] "RemoveContainer" containerID="0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5" Jul 24 14:17:25.027304 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:17:25.027285 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5\": container with ID starting with 0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5 not found: ID does not exist" containerID="0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5" Jul 24 14:17:25.027358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.027314 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5"} err="failed to get container status \"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5\": rpc error: code = NotFound desc = could not find container \"0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5\": container with ID starting with 0bf202b94a0d6e93c1fb9c7b2888f7a0583bd482a4f1516ffe37110e3d289bf5 not found: ID does not exist" Jul 24 14:17:25.027358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.027332 2581 scope.go:117] "RemoveContainer" containerID="358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1" Jul 24 14:17:25.037489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.037458 2581 scope.go:117] "RemoveContainer" containerID="358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1" Jul 24 14:17:25.037855 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:17:25.037834 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1\": container with ID starting with 358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1 not found: ID does not exist" containerID="358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1" Jul 24 14:17:25.037928 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.037864 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1"} err="failed to get container status \"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1\": rpc error: code = NotFound desc = could not find container \"358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1\": container with ID starting with 358573b74d19e486f3aa7b8e8d15b144fa36eaffaf52690d24b0827ab62117a1 not found: ID does not exist" Jul 24 14:17:25.046468 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.046441 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:17:25.059146 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059056 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume\") pod \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " Jul 24 14:17:25.059146 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059114 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg7ch\" (UniqueName: \"kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch\") pod \"744054df-ca54-4757-af37-28d3031db1b8\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " Jul 24 14:17:25.059367 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059165 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsb55\" (UniqueName: \"kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55\") pod \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\" (UID: \"de63b3e7-d07d-4498-8ee3-48490d6cb4db\") " Jul 24 14:17:25.059367 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059216 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume\") pod \"744054df-ca54-4757-af37-28d3031db1b8\" (UID: \"744054df-ca54-4757-af37-28d3031db1b8\") " Jul 24 14:17:25.059486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059381 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a9ea99-a507-4766-bbc5-480533689bdf-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.059486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059393 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rwr\" (UniqueName: \"kubernetes.io/projected/62a9ea99-a507-4766-bbc5-480533689bdf-kube-api-access-56rwr\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.059486 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059402 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62a9ea99-a507-4766-bbc5-480533689bdf-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.059690 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059469 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de63b3e7-d07d-4498-8ee3-48490d6cb4db" (UID: "de63b3e7-d07d-4498-8ee3-48490d6cb4db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:25.059744 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.059703 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "744054df-ca54-4757-af37-28d3031db1b8" (UID: "744054df-ca54-4757-af37-28d3031db1b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:25.061518 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.061491 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch" (OuterVolumeSpecName: "kube-api-access-bg7ch") pod "744054df-ca54-4757-af37-28d3031db1b8" (UID: "744054df-ca54-4757-af37-28d3031db1b8"). InnerVolumeSpecName "kube-api-access-bg7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:25.061730 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.061710 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55" (OuterVolumeSpecName: "kube-api-access-lsb55") pod "de63b3e7-d07d-4498-8ee3-48490d6cb4db" (UID: "de63b3e7-d07d-4498-8ee3-48490d6cb4db"). InnerVolumeSpecName "kube-api-access-lsb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:25.075326 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.075290 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6304ef-84d66b5c5c-spk25"] Jul 24 14:17:25.160496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.160463 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsb55\" (UniqueName: \"kubernetes.io/projected/de63b3e7-d07d-4498-8ee3-48490d6cb4db-kube-api-access-lsb55\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.160496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.160493 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744054df-ca54-4757-af37-28d3031db1b8-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.160496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.160508 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de63b3e7-d07d-4498-8ee3-48490d6cb4db-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.160798 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.160517 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7ch\" (UniqueName: \"kubernetes.io/projected/744054df-ca54-4757-af37-28d3031db1b8-kube-api-access-bg7ch\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:25.335359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.335323 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:17:25.340260 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.340225 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6304ef-5c6664b96-4cc8q"] Jul 24 14:17:25.350560 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.350526 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:17:25.352237 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:25.352214 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6304ef-6ffd5bc6fd-x4fwq"] Jul 24 14:17:26.917766 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:26.917731 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a9ea99-a507-4766-bbc5-480533689bdf" path="/var/lib/kubelet/pods/62a9ea99-a507-4766-bbc5-480533689bdf/volumes" Jul 24 14:17:26.918275 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:26.918252 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744054df-ca54-4757-af37-28d3031db1b8" path="/var/lib/kubelet/pods/744054df-ca54-4757-af37-28d3031db1b8/volumes" Jul 24 14:17:26.918701 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:26.918683 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" path="/var/lib/kubelet/pods/de63b3e7-d07d-4498-8ee3-48490d6cb4db/volumes" Jul 24 14:17:29.138894 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:29.138861 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:29.550213 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:29.550163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:35.658879 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:35.658823 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:35.761358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:35.761324 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:35.929122 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:35.929086 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:17:35.929364 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:35.929339 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" containerID="cri-o://d6f0f44d09432a6d039f3ed5906c54bc8fdc3bcb4b9e9ca996f5982d0f7c644a" gracePeriod=30 Jul 24 14:17:35.951096 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:35.951059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:36.279286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:36.279184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:41.096705 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.096676 2581 generic.go:356] "Generic (PLEG): container finished" podID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerID="d6f0f44d09432a6d039f3ed5906c54bc8fdc3bcb4b9e9ca996f5982d0f7c644a" exitCode=0 Jul 24 14:17:41.097066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.096725 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerDied","Data":"d6f0f44d09432a6d039f3ed5906c54bc8fdc3bcb4b9e9ca996f5982d0f7c644a"} Jul 24 14:17:41.097442 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.096775 2581 scope.go:117] "RemoveContainer" containerID="ea035b53d91e5986a74d297de6dcc4978638e9228a41429477be32342e4002df" Jul 24 14:17:41.191467 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.191402 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:17:41.220750 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220721 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220750 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220753 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220788 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220813 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsxl9\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220874 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220919 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.220957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220948 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.221270 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.220998 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket\") pod \"214e4513-a62d-4a6e-bb05-fd7baec0318c\" (UID: \"214e4513-a62d-4a6e-bb05-fd7baec0318c\") " Jul 24 14:17:41.221270 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221096 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:17:41.221389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221307 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:17:41.221389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221361 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.221499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221465 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:41.221558 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221533 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:17:41.222025 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.221989 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data" (OuterVolumeSpecName: "istio-data") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:17:41.223156 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.223126 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:17:41.223285 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.223269 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token" (OuterVolumeSpecName: "istio-token") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:41.223357 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.223340 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:17:41.223499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.223472 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9" (OuterVolumeSpecName: "kube-api-access-wsxl9") pod "214e4513-a62d-4a6e-bb05-fd7baec0318c" (UID: "214e4513-a62d-4a6e-bb05-fd7baec0318c"). InnerVolumeSpecName "kube-api-access-wsxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:41.321817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321773 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxl9\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-kube-api-access-wsxl9\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.321817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321808 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.321817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321819 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.321817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321828 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.322097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321837 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/214e4513-a62d-4a6e-bb05-fd7baec0318c-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.322097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321846 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/214e4513-a62d-4a6e-bb05-fd7baec0318c-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.322097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321853 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:41.322097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:41.321861 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/214e4513-a62d-4a6e-bb05-fd7baec0318c-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:42.102311 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.102285 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" Jul 24 14:17:42.102769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.102323 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp" event={"ID":"214e4513-a62d-4a6e-bb05-fd7baec0318c","Type":"ContainerDied","Data":"8be47a0512021da84be4e1d20201eb8287c85389cfb83ad86683c566bba1f30a"} Jul 24 14:17:42.102769 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.102391 2581 scope.go:117] "RemoveContainer" containerID="d6f0f44d09432a6d039f3ed5906c54bc8fdc3bcb4b9e9ca996f5982d0f7c644a" Jul 24 14:17:42.124635 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.124606 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:17:42.128319 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.128288 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6304ef-openshift-default-7f6499d5bb-mlnxp"] Jul 24 14:17:42.916137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:42.916098 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" path="/var/lib/kubelet/pods/214e4513-a62d-4a6e-bb05-fd7baec0318c/volumes" Jul 24 14:17:46.852667 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.852623 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:17:46.853315 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.852899 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" podUID="c54ed840-a4c5-4721-82e2-5c209c970183" containerName="maas-api" containerID="cri-o://fd563c726d4324517ef6ac942c791c7e95e063adccb171b06ed5276b8914c950" gracePeriod=30 Jul 24 14:17:46.887527 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.887486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:46.920734 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.920702 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:17:46.920988 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.920962 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" podUID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" containerName="payload-processing" containerID="cri-o://9c59234eba6c4e55f2302d3b026910ab41b1d9f833ca1a2cc9065d3b6cf02cdc" gracePeriod=30 Jul 24 14:17:46.998786 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:46.998756 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:47.133178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.133082 2581 generic.go:356] "Generic (PLEG): container finished" podID="c54ed840-a4c5-4721-82e2-5c209c970183" containerID="fd563c726d4324517ef6ac942c791c7e95e063adccb171b06ed5276b8914c950" exitCode=0 Jul 24 14:17:47.133178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.133151 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" event={"ID":"c54ed840-a4c5-4721-82e2-5c209c970183","Type":"ContainerDied","Data":"fd563c726d4324517ef6ac942c791c7e95e063adccb171b06ed5276b8914c950"} Jul 24 14:17:47.135078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.135057 2581 generic.go:356] "Generic (PLEG): container finished" podID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" containerID="9c59234eba6c4e55f2302d3b026910ab41b1d9f833ca1a2cc9065d3b6cf02cdc" exitCode=0 Jul 24 14:17:47.135191 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.135113 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" event={"ID":"3fec5c3d-f4dd-436c-be73-91a44db6ed51","Type":"ContainerDied","Data":"9c59234eba6c4e55f2302d3b026910ab41b1d9f833ca1a2cc9065d3b6cf02cdc"} Jul 24 14:17:47.197761 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.197732 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:17:47.220331 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.220299 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:17:47.270513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270479 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca\") pod \"c54ed840-a4c5-4721-82e2-5c209c970183\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " Jul 24 14:17:47.270718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270576 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls\") pod \"c54ed840-a4c5-4721-82e2-5c209c970183\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " Jul 24 14:17:47.270718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270634 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47snm\" (UniqueName: \"kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm\") pod \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " Jul 24 14:17:47.270718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270668 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th6nw\" (UniqueName: \"kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw\") pod \"c54ed840-a4c5-4721-82e2-5c209c970183\" (UID: \"c54ed840-a4c5-4721-82e2-5c209c970183\") " Jul 24 14:17:47.270718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270695 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume\") pod \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\" (UID: \"3fec5c3d-f4dd-436c-be73-91a44db6ed51\") " Jul 24 14:17:47.270943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.270928 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c54ed840-a4c5-4721-82e2-5c209c970183" (UID: "c54ed840-a4c5-4721-82e2-5c209c970183"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:47.271200 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.271172 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fec5c3d-f4dd-436c-be73-91a44db6ed51" (UID: "3fec5c3d-f4dd-436c-be73-91a44db6ed51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:47.272962 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.272932 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw" (OuterVolumeSpecName: "kube-api-access-th6nw") pod "c54ed840-a4c5-4721-82e2-5c209c970183" (UID: "c54ed840-a4c5-4721-82e2-5c209c970183"). InnerVolumeSpecName "kube-api-access-th6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:47.273293 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.273274 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm" (OuterVolumeSpecName: "kube-api-access-47snm") pod "3fec5c3d-f4dd-436c-be73-91a44db6ed51" (UID: "3fec5c3d-f4dd-436c-be73-91a44db6ed51"). InnerVolumeSpecName "kube-api-access-47snm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:47.273346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.273274 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c54ed840-a4c5-4721-82e2-5c209c970183" (UID: "c54ed840-a4c5-4721-82e2-5c209c970183"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:17:47.372053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.372011 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c54ed840-a4c5-4721-82e2-5c209c970183-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.372053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.372061 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47snm\" (UniqueName: \"kubernetes.io/projected/3fec5c3d-f4dd-436c-be73-91a44db6ed51-kube-api-access-47snm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.372322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.372076 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th6nw\" (UniqueName: \"kubernetes.io/projected/c54ed840-a4c5-4721-82e2-5c209c970183-kube-api-access-th6nw\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.372322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.372090 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fec5c3d-f4dd-436c-be73-91a44db6ed51-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.372322 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:47.372104 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54ed840-a4c5-4721-82e2-5c209c970183-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:17:48.141035 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.140945 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" Jul 24 14:17:48.141035 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.140974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d" event={"ID":"c54ed840-a4c5-4721-82e2-5c209c970183","Type":"ContainerDied","Data":"3f63ba4c21196635092419397e595c0c02daf03e5532f8c6e1992ad25088ef44"} Jul 24 14:17:48.141572 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.141039 2581 scope.go:117] "RemoveContainer" containerID="fd563c726d4324517ef6ac942c791c7e95e063adccb171b06ed5276b8914c950" Jul 24 14:17:48.142520 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.142500 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" Jul 24 14:17:48.142614 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.142502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42" event={"ID":"3fec5c3d-f4dd-436c-be73-91a44db6ed51","Type":"ContainerDied","Data":"a28ff69054fa17f41ef003c102a61a515b1c6ccee6f2884cb495d8708bef2e85"} Jul 24 14:17:48.151053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.151034 2581 scope.go:117] "RemoveContainer" containerID="9c59234eba6c4e55f2302d3b026910ab41b1d9f833ca1a2cc9065d3b6cf02cdc" Jul 24 14:17:48.164340 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.164309 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:17:48.168888 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.168853 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-406bb9-774cc45d95-7874d"] Jul 24 14:17:48.179779 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.179739 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:17:48.185501 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.185474 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-406bb9-68cc976578-bpt42"] Jul 24 14:17:48.916859 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.916817 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" path="/var/lib/kubelet/pods/3fec5c3d-f4dd-436c-be73-91a44db6ed51/volumes" Jul 24 14:17:48.917303 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:48.917283 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54ed840-a4c5-4721-82e2-5c209c970183" path="/var/lib/kubelet/pods/c54ed840-a4c5-4721-82e2-5c209c970183/volumes" Jul 24 14:17:51.439897 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:51.439865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:51.828751 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:51.828645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:57.868839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:57.868805 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:57.969165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:57.969122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:58.139767 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:58.139682 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:17:58.139986 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:58.139955 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" containerID="cri-o://ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84" gracePeriod=30 Jul 24 14:17:58.163292 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:58.163253 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:58.416888 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:58.416848 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:58.793390 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:58.793280 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:59.103284 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103201 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:17:59.103660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103615 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744054df-ca54-4757-af37-28d3031db1b8" containerName="payload-pre-processing" Jul 24 14:17:59.103660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103631 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="744054df-ca54-4757-af37-28d3031db1b8" containerName="payload-pre-processing" Jul 24 14:17:59.103660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103645 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" containerName="payload-processing" Jul 24 14:17:59.103660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103652 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" containerName="payload-processing" Jul 24 14:17:59.103660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103662 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103668 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103675 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103680 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103690 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a9ea99-a507-4766-bbc5-480533689bdf" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103698 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a9ea99-a507-4766-bbc5-480533689bdf" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103709 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" containerName="payload-processing" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103714 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" containerName="payload-processing" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103720 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54ed840-a4c5-4721-82e2-5c209c970183" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103725 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54ed840-a4c5-4721-82e2-5c209c970183" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103789 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103799 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fec5c3d-f4dd-436c-be73-91a44db6ed51" containerName="payload-processing" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103805 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a9ea99-a507-4766-bbc5-480533689bdf" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103812 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54ed840-a4c5-4721-82e2-5c209c970183" containerName="maas-api" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103817 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de63b3e7-d07d-4498-8ee3-48490d6cb4db" containerName="payload-processing" Jul 24 14:17:59.103835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.103823 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="744054df-ca54-4757-af37-28d3031db1b8" containerName="payload-pre-processing" Jul 24 14:17:59.108509 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.108490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.111179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.111157 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e05489-openshift-default-dockercfg-r8njg\"" Jul 24 14:17:59.123177 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.123148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:17:59.138771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.138704 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:59.182814 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182758 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fh2x\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.182971 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.183082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.183156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.183277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.183192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283650 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283727 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fh2x\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283742 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.283845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.283783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284079 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284159 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.284824 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.284803 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.286224 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.286183 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.286351 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.286334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.291734 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.291712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.291860 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.291780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fh2x\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x\") pod \"e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.322033 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.321994 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:17:59.419663 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.419633 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:17:59.566226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.566186 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:17:59.569510 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:17:59.569476 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25755a7_98c6_48ad_a940_163d08f5f256.slice/crio-6c90e86af5babd65f1466cf07263ac9413a52da6dfc2dfb98a413577e4bcd593 WatchSource:0}: Error finding container 6c90e86af5babd65f1466cf07263ac9413a52da6dfc2dfb98a413577e4bcd593: Status 404 returned error can't find the container with id 6c90e86af5babd65f1466cf07263ac9413a52da6dfc2dfb98a413577e4bcd593 Jul 24 14:17:59.571859 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.571819 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:17:59.571952 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.571907 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:17:59.571992 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:17:59.571947 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:00.206033 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:00.205993 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerStarted","Data":"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d"} Jul 24 14:18:00.206458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:00.206058 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerStarted","Data":"6c90e86af5babd65f1466cf07263ac9413a52da6dfc2dfb98a413577e4bcd593"} Jul 24 14:18:00.228839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:00.228789 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podStartSLOduration=1.228765539 podStartE2EDuration="1.228765539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:00.22631831 +0000 UTC m=+2373.822882379" watchObservedRunningTime="2026-07-24 14:18:00.228765539 +0000 UTC m=+2373.825329593" Jul 24 14:18:00.420453 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:00.420405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:00.425404 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:00.425377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:01.210449 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:01.210413 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:01.211300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:01.211279 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:02.343151 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.343116 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:02.343656 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.343580 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="214e4513-a62d-4a6e-bb05-fd7baec0318c" containerName="istio-proxy" Jul 24 14:18:02.346537 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.346519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.351373 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.351341 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e05489-serving-cert\"" Jul 24 14:18:02.351723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.351706 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:18:02.366832 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.366798 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:02.413608 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.413546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcp9\" (UniqueName: \"kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.413826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.413619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.413826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.413694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.514198 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.514158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcp9\" (UniqueName: \"kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.514408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.514209 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.514408 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.514321 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.514546 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:02.514419 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-serving-cert: secret "maas-api-e2e-api-a-e05489-serving-cert" not found Jul 24 14:18:02.514546 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:02.514501 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls podName:cc727441-36ae-4f3f-b0bb-4a02227e28b5 nodeName:}" failed. No retries permitted until 2026-07-24 14:18:03.014478266 +0000 UTC m=+2376.611042301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls") pod "maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" (UID: "cc727441-36ae-4f3f-b0bb-4a02227e28b5") : secret "maas-api-e2e-api-a-e05489-serving-cert" not found Jul 24 14:18:02.514954 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.514936 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.535448 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.535408 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcp9\" (UniqueName: \"kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:02.580480 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.580437 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:02.683331 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:02.683286 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:03.020656 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.020529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:03.023343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.023309 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") pod \"maas-api-e2e-api-a-e05489-76c9c78987-m8vzq\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:03.169295 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.169265 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:03.172792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.172767 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.176071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.176046 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e05489\"" Jul 24 14:18:03.176071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.176063 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e05489-dockercfg-szdqg\"" Jul 24 14:18:03.187741 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.187698 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:03.190250 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.190223 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:03.190712 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.190682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:03.193999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.193975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.202019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.201992 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:03.258763 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.258409 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:03.323085 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.323042 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk92n\" (UniqueName: \"kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.323276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.323095 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwph2\" (UniqueName: \"kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.323276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.323139 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.323276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.323242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.328748 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.328036 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:03.424059 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.424028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.424487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.424145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk92n\" (UniqueName: \"kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.424487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.424176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwph2\" (UniqueName: \"kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.424487 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.424211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.425206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.425053 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.425206 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.425105 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.435084 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.435048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk92n\" (UniqueName: \"kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n\") pod \"payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.435272 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.435136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwph2\" (UniqueName: \"kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2\") pod \"payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.436453 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.436429 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:18:03.455025 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.454887 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:03.457633 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:03.457567 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc727441_36ae_4f3f_b0bb_4a02227e28b5.slice/crio-ed3238fba43d3596fd824513beec9deb8640edff22c9f981a73d141d1a2e0c57 WatchSource:0}: Error finding container ed3238fba43d3596fd824513beec9deb8640edff22c9f981a73d141d1a2e0c57: Status 404 returned error can't find the container with id ed3238fba43d3596fd824513beec9deb8640edff22c9f981a73d141d1a2e0c57 Jul 24 14:18:03.459496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.459469 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:18:03.518775 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.518730 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:03.525090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525064 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525128 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525153 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525221 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525323 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525269 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525428 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525355 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525518 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525526 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525576 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525618 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.525845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525671 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo\") pod \"f127ffe8-5110-445c-9d5c-7240e03306d2\" (UID: \"f127ffe8-5110-445c-9d5c-7240e03306d2\") " Jul 24 14:18:03.526205 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525952 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:03.526205 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.525965 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:03.526205 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526099 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:03.526472 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526452 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.526555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526477 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.526555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526490 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.526555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526501 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f127ffe8-5110-445c-9d5c-7240e03306d2-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.526555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.526513 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.527782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.527751 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g" (OuterVolumeSpecName: "kube-api-access-rwm8g") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "kube-api-access-rwm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:03.527882 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.527849 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:03.528186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.528166 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:18:03.528234 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.528195 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f127ffe8-5110-445c-9d5c-7240e03306d2" (UID: "f127ffe8-5110-445c-9d5c-7240e03306d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:03.627808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.627182 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.627808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.627208 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/f127ffe8-5110-445c-9d5c-7240e03306d2-kube-api-access-rwm8g\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.627808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.627217 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.627808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.627227 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f127ffe8-5110-445c-9d5c-7240e03306d2-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:03.681983 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.681933 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:03.684815 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:03.684770 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7caeb5b_15e1_4c8e_b46d_5870e96ab793.slice/crio-9226fd22cd76adfc197d2921bcdd6b2ece1ab8c5f523f3c7aee3a8cb2991c042 WatchSource:0}: Error finding container 9226fd22cd76adfc197d2921bcdd6b2ece1ab8c5f523f3c7aee3a8cb2991c042: Status 404 returned error can't find the container with id 9226fd22cd76adfc197d2921bcdd6b2ece1ab8c5f523f3c7aee3a8cb2991c042 Jul 24 14:18:03.710168 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:03.710138 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:03.712117 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:03.712079 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2cd923_eefb_4361_9c88_95fcfa4b0717.slice/crio-22f46ef30664068d7d1aeb6429db1a3acc4c202ebf1eb381f02b05a4d0fc00d1 WatchSource:0}: Error finding container 22f46ef30664068d7d1aeb6429db1a3acc4c202ebf1eb381f02b05a4d0fc00d1: Status 404 returned error can't find the container with id 22f46ef30664068d7d1aeb6429db1a3acc4c202ebf1eb381f02b05a4d0fc00d1 Jul 24 14:18:04.230776 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.230739 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" event={"ID":"9f2cd923-eefb-4361-9c88-95fcfa4b0717","Type":"ContainerStarted","Data":"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d"} Jul 24 14:18:04.230997 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.230803 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" event={"ID":"9f2cd923-eefb-4361-9c88-95fcfa4b0717","Type":"ContainerStarted","Data":"22f46ef30664068d7d1aeb6429db1a3acc4c202ebf1eb381f02b05a4d0fc00d1"} Jul 24 14:18:04.230997 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.230937 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:04.232389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.232356 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" event={"ID":"c7caeb5b-15e1-4c8e-b46d-5870e96ab793","Type":"ContainerStarted","Data":"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa"} Jul 24 14:18:04.232531 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.232415 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" event={"ID":"c7caeb5b-15e1-4c8e-b46d-5870e96ab793","Type":"ContainerStarted","Data":"9226fd22cd76adfc197d2921bcdd6b2ece1ab8c5f523f3c7aee3a8cb2991c042"} Jul 24 14:18:04.232531 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.232445 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:04.233526 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.233504 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" event={"ID":"cc727441-36ae-4f3f-b0bb-4a02227e28b5","Type":"ContainerStarted","Data":"ed3238fba43d3596fd824513beec9deb8640edff22c9f981a73d141d1a2e0c57"} Jul 24 14:18:04.235085 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.235063 2581 generic.go:356] "Generic (PLEG): container finished" podID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerID="ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84" exitCode=0 Jul 24 14:18:04.235174 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.235097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerDied","Data":"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84"} Jul 24 14:18:04.235174 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.235109 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" Jul 24 14:18:04.235174 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.235138 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh" event={"ID":"f127ffe8-5110-445c-9d5c-7240e03306d2","Type":"ContainerDied","Data":"613e7f9761e0fae7ba078f2e38013e9086e46dd7a72a372d554396277af9729c"} Jul 24 14:18:04.235174 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.235167 2581 scope.go:117] "RemoveContainer" containerID="ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84" Jul 24 14:18:04.244507 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.244482 2581 scope.go:117] "RemoveContainer" containerID="f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c" Jul 24 14:18:04.249477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.249430 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" podStartSLOduration=1.249413725 podStartE2EDuration="1.249413725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:04.248987661 +0000 UTC m=+2377.845551807" watchObservedRunningTime="2026-07-24 14:18:04.249413725 +0000 UTC m=+2377.845977777" Jul 24 14:18:04.253926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.253901 2581 scope.go:117] "RemoveContainer" containerID="ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84" Jul 24 14:18:04.254215 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:04.254197 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84\": container with ID starting with ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84 not found: ID does not exist" containerID="ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84" Jul 24 14:18:04.254257 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.254223 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84"} err="failed to get container status \"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84\": rpc error: code = NotFound desc = could not find container \"ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84\": container with ID starting with ee249da8ffb0cd724b977355f6cad36bf318464140cb00a4269f72f704e17a84 not found: ID does not exist" Jul 24 14:18:04.254257 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.254238 2581 scope.go:117] "RemoveContainer" containerID="f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c" Jul 24 14:18:04.254511 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:04.254495 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c\": container with ID starting with f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c not found: ID does not exist" containerID="f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c" Jul 24 14:18:04.254568 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.254513 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c"} err="failed to get container status \"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c\": rpc error: code = NotFound desc = could not find container \"f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c\": container with ID starting with f3bbc0703a6346e13d88c0b9a02fa810a37760689c8dea6cc746439d3faf650c not found: ID does not exist" Jul 24 14:18:04.267543 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.267338 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" podStartSLOduration=1.267318124 podStartE2EDuration="1.267318124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:04.266452833 +0000 UTC m=+2377.863016895" watchObservedRunningTime="2026-07-24 14:18:04.267318124 +0000 UTC m=+2377.863882179" Jul 24 14:18:04.282111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.282074 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:18:04.286253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.286214 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-406bb9-openshift-default-858477c8c4-sm6nh"] Jul 24 14:18:04.919764 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:04.919735 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" path="/var/lib/kubelet/pods/f127ffe8-5110-445c-9d5c-7240e03306d2/volumes" Jul 24 14:18:05.241271 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:05.241237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" event={"ID":"cc727441-36ae-4f3f-b0bb-4a02227e28b5","Type":"ContainerStarted","Data":"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480"} Jul 24 14:18:05.241504 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:05.241297 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:05.262197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:05.262137 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" podStartSLOduration=1.8264636589999999 podStartE2EDuration="3.262119839s" totalImagesPullingTime="1.43565618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:05.260259495 +0000 UTC m=+2378.856823554" watchObservedRunningTime="2026-07-24 14:18:05.262119839 +0000 UTC m=+2378.858683895" Jul 24 14:18:06.776890 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:06.776858 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:07.303935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.303882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:07.630879 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.630802 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:18:07.631256 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631241 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.631346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631258 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.631346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631284 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.631346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631290 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.631524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631350 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.631524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.631358 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f127ffe8-5110-445c-9d5c-7240e03306d2" containerName="istio-proxy" Jul 24 14:18:07.634764 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.634726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.637148 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.637128 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-517d91-openshift-default-dockercfg-782b4\"" Jul 24 14:18:07.645771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.645727 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:18:07.661241 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.661205 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:07.766005 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.765968 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772v9\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766533 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.766533 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.766430 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867300 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867338 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867410 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772v9\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.867843 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.868263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867850 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.868263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867962 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.868263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.867994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.868263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.868201 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.868576 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.868555 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.870446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.870410 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.870446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.870431 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.876078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.876046 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.876718 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.876697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772v9\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9\") pod \"e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:07.921437 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.921402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:07.949337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:07.949300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:08.101511 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.101485 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:18:08.103259 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:08.103231 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e213060_abc5_4005_9cb2_82b7cd7b99f4.slice/crio-201348067e5237ba980366fa6c3c9723c9a70056789d11d34733f47a6353f932 WatchSource:0}: Error finding container 201348067e5237ba980366fa6c3c9723c9a70056789d11d34733f47a6353f932: Status 404 returned error can't find the container with id 201348067e5237ba980366fa6c3c9723c9a70056789d11d34733f47a6353f932 Jul 24 14:18:08.105790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.105752 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:08.105899 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.105868 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:08.105938 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.105908 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:08.262244 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.262209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerStarted","Data":"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59"} Jul 24 14:18:08.262425 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.262269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerStarted","Data":"201348067e5237ba980366fa6c3c9723c9a70056789d11d34733f47a6353f932"} Jul 24 14:18:08.284260 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.284197 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podStartSLOduration=1.284173711 podStartE2EDuration="1.284173711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:08.28107969 +0000 UTC m=+2381.877643746" watchObservedRunningTime="2026-07-24 14:18:08.284173711 +0000 UTC m=+2381.880737769" Jul 24 14:18:08.949915 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:08.949876 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:09.955683 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:09.955653 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:10.276129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.275997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:10.277348 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.277312 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:10.283218 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.283186 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:10.415509 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.415458 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:18:10.419197 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.419170 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:10.419358 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.419209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.422170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.422138 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-517d91-serving-cert\"" Jul 24 14:18:10.443579 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.443537 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:18:10.495803 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.495762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.495990 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.495852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qxz\" (UniqueName: \"kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.495990 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.495905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.523309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.523270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:10.597378 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.597283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.597378 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.597356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qxz\" (UniqueName: \"kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.597620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.597397 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.597620 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:10.597462 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-serving-cert: secret "maas-api-e2e-api-b-517d91-serving-cert" not found Jul 24 14:18:10.597620 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:10.597546 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls podName:65dc73e3-213a-44ec-8a32-58530f5ec1f6 nodeName:}" failed. No retries permitted until 2026-07-24 14:18:11.097524802 +0000 UTC m=+2384.694088843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls") pod "maas-api-e2e-api-b-517d91-6b95db496-57vsh" (UID: "65dc73e3-213a-44ec-8a32-58530f5ec1f6") : secret "maas-api-e2e-api-b-517d91-serving-cert" not found Jul 24 14:18:10.598069 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.598049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.606225 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.606190 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qxz\" (UniqueName: \"kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:10.735186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:10.735154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:11.048265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.048235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:11.105777 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.103622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:11.108918 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.108880 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") pod \"maas-api-e2e-api-b-517d91-6b95db496-57vsh\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:11.148261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.148223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:11.186620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.186402 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:18:11.192821 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.191600 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.195960 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.195930 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-517d91-dockercfg-94drh\"" Jul 24 14:18:11.195960 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.195946 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-517d91\"" Jul 24 14:18:11.200266 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.200232 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:18:11.254428 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.254401 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:11.316346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.311348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.316346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.311431 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nps\" (UniqueName: \"kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.331632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.331575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:11.412917 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.412647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nps\" (UniqueName: \"kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.412917 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.412890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.414768 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.414313 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.421749 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.421667 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nps\" (UniqueName: \"kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps\") pod \"payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.507956 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.507909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:11.546330 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.546228 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:18:11.547407 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:11.547354 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dc73e3_213a_44ec_8a32_58530f5ec1f6.slice/crio-625d1456bdaa47207726c25261fe44cf5b694a34b93e302e68f023940d3670e0 WatchSource:0}: Error finding container 625d1456bdaa47207726c25261fe44cf5b694a34b93e302e68f023940d3670e0: Status 404 returned error can't find the container with id 625d1456bdaa47207726c25261fe44cf5b694a34b93e302e68f023940d3670e0 Jul 24 14:18:11.728391 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:11.728356 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:18:11.731629 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:18:11.731595 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f89d3d_6006_41ac_95cf_7e694d77e7e5.slice/crio-81f86a8c40570cb1733ab518f729bbc77fb350e3857c70a0ced41087b900b47f WatchSource:0}: Error finding container 81f86a8c40570cb1733ab518f729bbc77fb350e3857c70a0ced41087b900b47f: Status 404 returned error can't find the container with id 81f86a8c40570cb1733ab518f729bbc77fb350e3857c70a0ced41087b900b47f Jul 24 14:18:12.321333 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.321300 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" event={"ID":"51f89d3d-6006-41ac-95cf-7e694d77e7e5","Type":"ContainerStarted","Data":"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec"} Jul 24 14:18:12.321806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.321357 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" event={"ID":"51f89d3d-6006-41ac-95cf-7e694d77e7e5","Type":"ContainerStarted","Data":"81f86a8c40570cb1733ab518f729bbc77fb350e3857c70a0ced41087b900b47f"} Jul 24 14:18:12.321806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.321381 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:12.322907 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.322881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" event={"ID":"65dc73e3-213a-44ec-8a32-58530f5ec1f6","Type":"ContainerStarted","Data":"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde"} Jul 24 14:18:12.323026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.322927 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" event={"ID":"65dc73e3-213a-44ec-8a32-58530f5ec1f6","Type":"ContainerStarted","Data":"625d1456bdaa47207726c25261fe44cf5b694a34b93e302e68f023940d3670e0"} Jul 24 14:18:12.323114 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.323096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:12.339493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.339431 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" podStartSLOduration=1.3394115420000001 podStartE2EDuration="1.339411542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:12.338115417 +0000 UTC m=+2385.934679475" watchObservedRunningTime="2026-07-24 14:18:12.339411542 +0000 UTC m=+2385.935975656" Jul 24 14:18:12.356497 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:12.356442 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" podStartSLOduration=2.018291549 podStartE2EDuration="2.356421449s" totalImagesPullingTime="338.1299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:12.355683723 +0000 UTC m=+2385.952247779" watchObservedRunningTime="2026-07-24 14:18:12.356421449 +0000 UTC m=+2385.952985503" Jul 24 14:18:14.831709 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:14.831672 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:15.244494 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:15.244472 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:15.244720 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:15.244621 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:17.252823 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:17.252775 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:18:17.253274 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:17.252845 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:18:18.334484 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.334453 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:18:18.359202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.359168 2581 generic.go:356] "Generic (PLEG): container finished" podID="a25755a7-98c6-48ad-a940-163d08f5f256" containerID="36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d" exitCode=0 Jul 24 14:18:18.359402 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.359229 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerDied","Data":"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d"} Jul 24 14:18:18.359814 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.359796 2581 scope.go:117] "RemoveContainer" containerID="36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d" Jul 24 14:18:18.361140 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.361108 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:18.361227 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.361197 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:18.361268 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:18.361242 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:19.367854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:19.367802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerStarted","Data":"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326"} Jul 24 14:18:19.420219 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:19.420128 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:19.420479 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:19.420061 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 24 14:18:19.420684 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:19.420660 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 24 14:18:20.421088 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:20.421046 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 24 14:18:20.421632 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:20.421104 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 24 14:18:21.420532 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:21.420486 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 24 14:18:21.420746 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:21.420561 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 24 14:18:22.420895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:22.420861 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 24 14:18:22.421386 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:22.421359 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 24 14:18:23.331768 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:23.331735 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:18:23.423149 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:23.423114 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Jul 24 14:18:23.423835 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:23.423780 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Jul 24 14:18:24.423966 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:24.423933 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:24.424435 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:24.424211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:24.424983 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:24.424967 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:26.317782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:26.317740 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 14:18:26.318182 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:26.317838 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 14:18:27.228702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:27.228672 2581 scope.go:117] "RemoveContainer" containerID="1df6521bba3c358cea169b0cee07d1bf2cf9994eebea7cd4df3b67a7d2f34b05" Jul 24 14:18:28.446192 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.446154 2581 generic.go:356] "Generic (PLEG): container finished" podID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerID="28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59" exitCode=0 Jul 24 14:18:28.446685 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.446227 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerDied","Data":"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59"} Jul 24 14:18:28.446740 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.446700 2581 scope.go:117] "RemoveContainer" containerID="28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59" Jul 24 14:18:28.447766 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.447735 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:28.447885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.447820 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:28.447885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:28.447867 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:18:29.453596 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:29.453560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerStarted","Data":"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923"} Jul 24 14:18:29.949478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:29.949442 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:29.949772 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:29.949749 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 24 14:18:29.949864 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:29.949804 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 24 14:18:30.950495 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:30.950455 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 24 14:18:30.950968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:30.950527 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 24 14:18:31.949825 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:31.949787 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 24 14:18:31.950042 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:31.949848 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 24 14:18:32.949903 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:32.949861 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Jul 24 14:18:32.950381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:32.949935 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Jul 24 14:18:33.953634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:33.953599 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:33.954019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:33.953987 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:33.954626 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:33.954605 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:18:38.962398 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:38.962365 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:38.962896 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:38.962746 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" podUID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" containerName="maas-api" containerID="cri-o://5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480" gracePeriod=30 Jul 24 14:18:38.991057 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:38.990998 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:39.021160 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.021121 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:39.021472 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.021420 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" podUID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" containerName="payload-processing" containerID="cri-o://112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d" gracePeriod=30 Jul 24 14:18:39.026023 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.025791 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:39.026851 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.026452 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" podUID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" containerName="payload-pre-processing" containerID="cri-o://09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa" gracePeriod=30 Jul 24 14:18:39.065436 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.065370 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:39.132866 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.132833 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:39.334861 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.334833 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:39.337943 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.337919 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:39.342000 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.341970 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:39.387436 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387401 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca\") pod \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387468 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk92n\" (UniqueName: \"kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n\") pod \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387532 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume\") pod \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387595 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume\") pod \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\" (UID: \"9f2cd923-eefb-4361-9c88-95fcfa4b0717\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387637 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwph2\" (UniqueName: \"kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2\") pod \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\" (UID: \"c7caeb5b-15e1-4c8e-b46d-5870e96ab793\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387680 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") pod \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.387739 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzcp9\" (UniqueName: \"kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9\") pod \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\" (UID: \"cc727441-36ae-4f3f-b0bb-4a02227e28b5\") " Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.388436 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc727441-36ae-4f3f-b0bb-4a02227e28b5" (UID: "cc727441-36ae-4f3f-b0bb-4a02227e28b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.388529 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7caeb5b-15e1-4c8e-b46d-5870e96ab793" (UID: "c7caeb5b-15e1-4c8e-b46d-5870e96ab793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:39.390492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.388565 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f2cd923-eefb-4361-9c88-95fcfa4b0717" (UID: "9f2cd923-eefb-4361-9c88-95fcfa4b0717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:39.391258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.390675 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc727441-36ae-4f3f-b0bb-4a02227e28b5" (UID: "cc727441-36ae-4f3f-b0bb-4a02227e28b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:18:39.391258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.390994 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2" (OuterVolumeSpecName: "kube-api-access-rwph2") pod "c7caeb5b-15e1-4c8e-b46d-5870e96ab793" (UID: "c7caeb5b-15e1-4c8e-b46d-5870e96ab793"). InnerVolumeSpecName "kube-api-access-rwph2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:39.391258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.391090 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n" (OuterVolumeSpecName: "kube-api-access-wk92n") pod "9f2cd923-eefb-4361-9c88-95fcfa4b0717" (UID: "9f2cd923-eefb-4361-9c88-95fcfa4b0717"). InnerVolumeSpecName "kube-api-access-wk92n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:39.391701 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.391678 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9" (OuterVolumeSpecName: "kube-api-access-nzcp9") pod "cc727441-36ae-4f3f-b0bb-4a02227e28b5" (UID: "cc727441-36ae-4f3f-b0bb-4a02227e28b5"). InnerVolumeSpecName "kube-api-access-nzcp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488836 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488872 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cd923-eefb-4361-9c88-95fcfa4b0717-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488881 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwph2\" (UniqueName: \"kubernetes.io/projected/c7caeb5b-15e1-4c8e-b46d-5870e96ab793-kube-api-access-rwph2\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488890 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc727441-36ae-4f3f-b0bb-4a02227e28b5-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488901 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcp9\" (UniqueName: \"kubernetes.io/projected/cc727441-36ae-4f3f-b0bb-4a02227e28b5-kube-api-access-nzcp9\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488909 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc727441-36ae-4f3f-b0bb-4a02227e28b5-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.488933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.488917 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk92n\" (UniqueName: \"kubernetes.io/projected/9f2cd923-eefb-4361-9c88-95fcfa4b0717-kube-api-access-wk92n\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:39.510552 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.510513 2581 generic.go:356] "Generic (PLEG): container finished" podID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" containerID="5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480" exitCode=0 Jul 24 14:18:39.510778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.510560 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" Jul 24 14:18:39.510778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.510580 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" event={"ID":"cc727441-36ae-4f3f-b0bb-4a02227e28b5","Type":"ContainerDied","Data":"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480"} Jul 24 14:18:39.510778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.510665 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq" event={"ID":"cc727441-36ae-4f3f-b0bb-4a02227e28b5","Type":"ContainerDied","Data":"ed3238fba43d3596fd824513beec9deb8640edff22c9f981a73d141d1a2e0c57"} Jul 24 14:18:39.510778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.510696 2581 scope.go:117] "RemoveContainer" containerID="5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480" Jul 24 14:18:39.512172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.512143 2581 generic.go:356] "Generic (PLEG): container finished" podID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" containerID="112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d" exitCode=0 Jul 24 14:18:39.512301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.512173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" event={"ID":"9f2cd923-eefb-4361-9c88-95fcfa4b0717","Type":"ContainerDied","Data":"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d"} Jul 24 14:18:39.512301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.512193 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" Jul 24 14:18:39.512301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.512221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh" event={"ID":"9f2cd923-eefb-4361-9c88-95fcfa4b0717","Type":"ContainerDied","Data":"22f46ef30664068d7d1aeb6429db1a3acc4c202ebf1eb381f02b05a4d0fc00d1"} Jul 24 14:18:39.513743 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.513716 2581 generic.go:356] "Generic (PLEG): container finished" podID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" containerID="09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa" exitCode=0 Jul 24 14:18:39.513875 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.513759 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" Jul 24 14:18:39.513875 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.513785 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" event={"ID":"c7caeb5b-15e1-4c8e-b46d-5870e96ab793","Type":"ContainerDied","Data":"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa"} Jul 24 14:18:39.513875 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.513828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt" event={"ID":"c7caeb5b-15e1-4c8e-b46d-5870e96ab793","Type":"ContainerDied","Data":"9226fd22cd76adfc197d2921bcdd6b2ece1ab8c5f523f3c7aee3a8cb2991c042"} Jul 24 14:18:39.522809 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.522768 2581 scope.go:117] "RemoveContainer" containerID="5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480" Jul 24 14:18:39.523162 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:39.523142 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480\": container with ID starting with 5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480 not found: ID does not exist" containerID="5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480" Jul 24 14:18:39.523222 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.523171 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480"} err="failed to get container status \"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480\": rpc error: code = NotFound desc = could not find container \"5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480\": container with ID starting with 5d6c791ee870340bc22cc9e53ae7c6b7f6382b0d6ac7edf2385b1bd412757480 not found: ID does not exist" Jul 24 14:18:39.523222 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.523195 2581 scope.go:117] "RemoveContainer" containerID="112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d" Jul 24 14:18:39.532150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.532128 2581 scope.go:117] "RemoveContainer" containerID="112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d" Jul 24 14:18:39.532488 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:39.532470 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d\": container with ID starting with 112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d not found: ID does not exist" containerID="112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d" Jul 24 14:18:39.532549 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.532497 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d"} err="failed to get container status \"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d\": rpc error: code = NotFound desc = could not find container \"112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d\": container with ID starting with 112f8a80ad0d32f35337cff0e736e7e3e8b2fb58f175e93cf368af110fddfc9d not found: ID does not exist" Jul 24 14:18:39.532549 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.532515 2581 scope.go:117] "RemoveContainer" containerID="09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa" Jul 24 14:18:39.535910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.535884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:39.542704 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.542679 2581 scope.go:117] "RemoveContainer" containerID="09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa" Jul 24 14:18:39.542902 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.542886 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e05489-76c9c78987-m8vzq"] Jul 24 14:18:39.543057 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:39.543036 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa\": container with ID starting with 09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa not found: ID does not exist" containerID="09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa" Jul 24 14:18:39.543103 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.543067 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa"} err="failed to get container status \"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa\": rpc error: code = NotFound desc = could not find container \"09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa\": container with ID starting with 09d5bd002ba8ad16b2245cee2bdb045f2dbff1356ced0c5c1e44bb32fb7c04fa not found: ID does not exist" Jul 24 14:18:39.552233 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.552192 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:39.555762 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.555731 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e05489-fc7474db8-h54zt"] Jul 24 14:18:39.566078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.566033 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:39.569261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:39.569225 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e05489-54fd6d7498-xmrfh"] Jul 24 14:18:40.916782 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:40.916751 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" path="/var/lib/kubelet/pods/9f2cd923-eefb-4361-9c88-95fcfa4b0717/volumes" Jul 24 14:18:40.917208 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:40.917143 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" path="/var/lib/kubelet/pods/c7caeb5b-15e1-4c8e-b46d-5870e96ab793/volumes" Jul 24 14:18:40.917494 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:40.917480 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" path="/var/lib/kubelet/pods/cc727441-36ae-4f3f-b0bb-4a02227e28b5/volumes" Jul 24 14:18:43.187522 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:43.187476 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:43.595379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:43.595296 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:49.526826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:49.526727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:49.594275 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:49.594226 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:49.827409 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:49.827313 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:18:49.827643 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:49.827615 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" containerID="cri-o://eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326" gracePeriod=30 Jul 24 14:18:49.848987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:49.848953 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:50.097342 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:50.097251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:18:54.426505 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:54.426472 2581 patch_prober.go:28] interesting pod/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 14:18:54.426912 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:54.426523 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 14:18:55.084231 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.084208 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:55.230497 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230462 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230497 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230504 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230799 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230523 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fh2x\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230799 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230567 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230799 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230637 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230826 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230865 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:55.230958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230924 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.230958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230952 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.231178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.230996 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket\") pod \"a25755a7-98c6-48ad-a940-163d08f5f256\" (UID: \"a25755a7-98c6-48ad-a940-163d08f5f256\") " Jul 24 14:18:55.231178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231008 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:55.231296 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231268 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25755a7-98c6-48ad-a940-163d08f5f256-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.231296 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231289 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.231404 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231313 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:55.231534 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231487 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:55.231724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.231702 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data" (OuterVolumeSpecName: "istio-data") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:55.232866 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.232846 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token" (OuterVolumeSpecName: "istio-token") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:55.232979 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.232958 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:18:55.233208 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.233188 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:18:55.233208 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.233200 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x" (OuterVolumeSpecName: "kube-api-access-9fh2x") pod "a25755a7-98c6-48ad-a940-163d08f5f256" (UID: "a25755a7-98c6-48ad-a940-163d08f5f256"). InnerVolumeSpecName "kube-api-access-9fh2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:18:55.331742 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331703 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331742 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331737 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331742 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331746 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331759 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fh2x\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-kube-api-access-9fh2x\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331771 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25755a7-98c6-48ad-a940-163d08f5f256-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331780 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25755a7-98c6-48ad-a940-163d08f5f256-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.331980 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.331788 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25755a7-98c6-48ad-a940-163d08f5f256-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:18:55.602634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.602530 2581 generic.go:356] "Generic (PLEG): container finished" podID="a25755a7-98c6-48ad-a940-163d08f5f256" containerID="eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326" exitCode=0 Jul 24 14:18:55.602634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.602598 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" Jul 24 14:18:55.602634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.602617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerDied","Data":"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326"} Jul 24 14:18:55.603147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.602682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn" event={"ID":"a25755a7-98c6-48ad-a940-163d08f5f256","Type":"ContainerDied","Data":"6c90e86af5babd65f1466cf07263ac9413a52da6dfc2dfb98a413577e4bcd593"} Jul 24 14:18:55.603147 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.602715 2581 scope.go:117] "RemoveContainer" containerID="eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326" Jul 24 14:18:55.612730 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.612711 2581 scope.go:117] "RemoveContainer" containerID="36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d" Jul 24 14:18:55.621467 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.621445 2581 scope.go:117] "RemoveContainer" containerID="eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326" Jul 24 14:18:55.621813 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:55.621794 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326\": container with ID starting with eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326 not found: ID does not exist" containerID="eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326" Jul 24 14:18:55.621869 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.621823 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326"} err="failed to get container status \"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326\": rpc error: code = NotFound desc = could not find container \"eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326\": container with ID starting with eb47596e9a541f277a84cac72bca8bf114b86ce0ab4f7be28936a4de1b7c7326 not found: ID does not exist" Jul 24 14:18:55.621869 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.621844 2581 scope.go:117] "RemoveContainer" containerID="36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d" Jul 24 14:18:55.622083 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:18:55.622068 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d\": container with ID starting with 36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d not found: ID does not exist" containerID="36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d" Jul 24 14:18:55.622126 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.622088 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d"} err="failed to get container status \"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d\": rpc error: code = NotFound desc = could not find container \"36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d\": container with ID starting with 36a5ecc369af70e976cf969ff4138ac81c4b4475b24c2442b987aab299d9952d not found: ID does not exist" Jul 24 14:18:55.625514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.625485 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:18:55.629098 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:55.629072 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e05489-openshift-default-79dc9cb9fd-btqdn"] Jul 24 14:18:56.916546 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:18:56.916510 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" path="/var/lib/kubelet/pods/a25755a7-98c6-48ad-a940-163d08f5f256/volumes" Jul 24 14:19:06.091757 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.091710 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:19:06.092370 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.091965 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" podUID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" containerName="maas-api" containerID="cri-o://678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde" gracePeriod=30 Jul 24 14:19:06.127391 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.127345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:06.152424 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.152381 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:19:06.152712 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.152680 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" podUID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" containerName="payload-processing" containerID="cri-o://9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec" gracePeriod=30 Jul 24 14:19:06.227374 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.227333 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:06.332451 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.332349 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:06.441995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.441968 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:19:06.443923 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.443903 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:19:06.531261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531223 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume\") pod \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " Jul 24 14:19:06.531456 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531282 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8nps\" (UniqueName: \"kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps\") pod \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\" (UID: \"51f89d3d-6006-41ac-95cf-7e694d77e7e5\") " Jul 24 14:19:06.531456 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531320 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7qxz\" (UniqueName: \"kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz\") pod \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " Jul 24 14:19:06.531456 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531357 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca\") pod \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " Jul 24 14:19:06.531456 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531407 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") pod \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\" (UID: \"65dc73e3-213a-44ec-8a32-58530f5ec1f6\") " Jul 24 14:19:06.531762 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531738 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51f89d3d-6006-41ac-95cf-7e694d77e7e5" (UID: "51f89d3d-6006-41ac-95cf-7e694d77e7e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:19:06.531830 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.531787 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65dc73e3-213a-44ec-8a32-58530f5ec1f6" (UID: "65dc73e3-213a-44ec-8a32-58530f5ec1f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:19:06.533627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.533546 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65dc73e3-213a-44ec-8a32-58530f5ec1f6" (UID: "65dc73e3-213a-44ec-8a32-58530f5ec1f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:19:06.533627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.533562 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz" (OuterVolumeSpecName: "kube-api-access-c7qxz") pod "65dc73e3-213a-44ec-8a32-58530f5ec1f6" (UID: "65dc73e3-213a-44ec-8a32-58530f5ec1f6"). InnerVolumeSpecName "kube-api-access-c7qxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:19:06.533627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.533601 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps" (OuterVolumeSpecName: "kube-api-access-f8nps") pod "51f89d3d-6006-41ac-95cf-7e694d77e7e5" (UID: "51f89d3d-6006-41ac-95cf-7e694d77e7e5"). InnerVolumeSpecName "kube-api-access-f8nps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:19:06.633138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.633047 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nps\" (UniqueName: \"kubernetes.io/projected/51f89d3d-6006-41ac-95cf-7e694d77e7e5-kube-api-access-f8nps\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.633138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.633079 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qxz\" (UniqueName: \"kubernetes.io/projected/65dc73e3-213a-44ec-8a32-58530f5ec1f6-kube-api-access-c7qxz\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.633138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.633089 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc73e3-213a-44ec-8a32-58530f5ec1f6-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.633138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.633098 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65dc73e3-213a-44ec-8a32-58530f5ec1f6-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.633138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.633112 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f89d3d-6006-41ac-95cf-7e694d77e7e5-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.660747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.660705 2581 generic.go:356] "Generic (PLEG): container finished" podID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" containerID="9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec" exitCode=0 Jul 24 14:19:06.660949 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.660778 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" Jul 24 14:19:06.660949 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.660789 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" event={"ID":"51f89d3d-6006-41ac-95cf-7e694d77e7e5","Type":"ContainerDied","Data":"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec"} Jul 24 14:19:06.660949 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.660879 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p" event={"ID":"51f89d3d-6006-41ac-95cf-7e694d77e7e5","Type":"ContainerDied","Data":"81f86a8c40570cb1733ab518f729bbc77fb350e3857c70a0ced41087b900b47f"} Jul 24 14:19:06.660949 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.660916 2581 scope.go:117] "RemoveContainer" containerID="9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec" Jul 24 14:19:06.662207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.662187 2581 generic.go:356] "Generic (PLEG): container finished" podID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" containerID="678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde" exitCode=0 Jul 24 14:19:06.662296 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.662262 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" Jul 24 14:19:06.662348 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.662261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" event={"ID":"65dc73e3-213a-44ec-8a32-58530f5ec1f6","Type":"ContainerDied","Data":"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde"} Jul 24 14:19:06.662403 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.662383 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh" event={"ID":"65dc73e3-213a-44ec-8a32-58530f5ec1f6","Type":"ContainerDied","Data":"625d1456bdaa47207726c25261fe44cf5b694a34b93e302e68f023940d3670e0"} Jul 24 14:19:06.672982 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.672957 2581 scope.go:117] "RemoveContainer" containerID="9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec" Jul 24 14:19:06.673367 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:06.673345 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec\": container with ID starting with 9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec not found: ID does not exist" containerID="9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec" Jul 24 14:19:06.673460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.673373 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec"} err="failed to get container status \"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec\": rpc error: code = NotFound desc = could not find container \"9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec\": container with ID starting with 9ba0f23339319c8840eb624242c3aabc1fa5d13e3c8b1bb7ad45ef17325438ec not found: ID does not exist" Jul 24 14:19:06.673460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.673395 2581 scope.go:117] "RemoveContainer" containerID="678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde" Jul 24 14:19:06.682014 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.681996 2581 scope.go:117] "RemoveContainer" containerID="678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde" Jul 24 14:19:06.682325 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:06.682299 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde\": container with ID starting with 678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde not found: ID does not exist" containerID="678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde" Jul 24 14:19:06.682401 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.682333 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde"} err="failed to get container status \"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde\": rpc error: code = NotFound desc = could not find container \"678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde\": container with ID starting with 678ad2861bbe120d4ffaa29130274335f5094efa95fc89557cf8cd1a6b9efcde not found: ID does not exist" Jul 24 14:19:06.712470 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.712439 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:19:06.717724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.717694 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-517d91-6b95db496-57vsh"] Jul 24 14:19:06.729498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.729459 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:19:06.733792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.733759 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-517d91-bcdb45764-5ng2p"] Jul 24 14:19:06.916674 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.916636 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" path="/var/lib/kubelet/pods/51f89d3d-6006-41ac-95cf-7e694d77e7e5/volumes" Jul 24 14:19:06.917115 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:06.917096 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" path="/var/lib/kubelet/pods/65dc73e3-213a-44ec-8a32-58530f5ec1f6/volumes" Jul 24 14:19:10.299447 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:10.299414 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:10.798900 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:10.798867 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:16.969982 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:16.969944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:16.996012 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:16.995979 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:19:16.996243 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:16.996219 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" containerID="cri-o://782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923" gracePeriod=30 Jul 24 14:19:17.038555 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.038502 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:17.269533 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.269423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:17.627895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.627797 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:17.941210 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941173 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:19:17.941538 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941525 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" containerName="maas-api" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941540 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" containerName="maas-api" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941551 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" containerName="payload-processing" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941557 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" containerName="payload-processing" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941566 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" containerName="payload-pre-processing" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941572 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" containerName="payload-pre-processing" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941579 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" containerName="payload-processing" Jul 24 14:19:17.941618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941618 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" containerName="payload-processing" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941630 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941636 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941645 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941650 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941661 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" containerName="maas-api" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941666 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" containerName="maas-api" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941720 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc727441-36ae-4f3f-b0bb-4a02227e28b5" containerName="maas-api" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941727 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f89d3d-6006-41ac-95cf-7e694d77e7e5" containerName="payload-processing" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941733 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941741 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7caeb5b-15e1-4c8e-b46d-5870e96ab793" containerName="payload-pre-processing" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941749 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2cd923-eefb-4361-9c88-95fcfa4b0717" containerName="payload-processing" Jul 24 14:19:17.941967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.941755 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dc73e3-213a-44ec-8a32-58530f5ec1f6" containerName="maas-api" Jul 24 14:19:17.946283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.946261 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:17.948626 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.948609 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a26cce-openshift-default-dockercfg-x7g76\"" Jul 24 14:19:17.956139 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.956110 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:19:17.971705 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:17.971650 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:18.034656 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnd9\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.034854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.034854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034705 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.034854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.034854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.034854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.035042 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.035042 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.035042 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.034961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.085363 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.085312 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:18.135643 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135561 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135727 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnd9\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.135838 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.135834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136019 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136075 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136130 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136182 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.136519 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.136496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.138343 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.138323 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.138565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.138550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.144103 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.144067 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.144251 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.144184 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnd9\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9\") pod \"e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.258634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.258508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:18.410969 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.410942 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:19:18.412965 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:18.412930 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca6835b_bc1b_4193_bd32_d76e96f7a26f.slice/crio-ee240e338c7068a233d718fed3d4655f3fe08f6bdc06ae6fc954ced35f046318 WatchSource:0}: Error finding container ee240e338c7068a233d718fed3d4655f3fe08f6bdc06ae6fc954ced35f046318: Status 404 returned error can't find the container with id ee240e338c7068a233d718fed3d4655f3fe08f6bdc06ae6fc954ced35f046318 Jul 24 14:19:18.415129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.415093 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:18.415250 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.415182 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:18.415250 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.415228 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:18.725683 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.725641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerStarted","Data":"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9"} Jul 24 14:19:18.725910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.725708 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerStarted","Data":"ee240e338c7068a233d718fed3d4655f3fe08f6bdc06ae6fc954ced35f046318"} Jul 24 14:19:18.748568 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.748506 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podStartSLOduration=1.748487566 podStartE2EDuration="1.748487566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:18.746975631 +0000 UTC m=+2452.343539686" watchObservedRunningTime="2026-07-24 14:19:18.748487566 +0000 UTC m=+2452.345051624" Jul 24 14:19:18.956766 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.956731 2581 patch_prober.go:28] interesting pod/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 14:19:18.956939 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:18.956786 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 14:19:19.258639 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:19.258600 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:19.264235 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:19.264206 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:19.730903 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:19.730863 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:19.731826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:19.731803 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:20.744152 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.744113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:20.865030 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.865000 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:20.960804 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.960773 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:19:20.961368 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.961355 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25755a7-98c6-48ad-a940-163d08f5f256" containerName="istio-proxy" Jul 24 14:19:20.964752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.964729 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:20.967335 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.967317 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a26cce-serving-cert\"" Jul 24 14:19:20.967785 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.967768 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 14:19:20.978822 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.978802 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:19:20.998662 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:20.998544 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:21.066968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.066933 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.066968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.066967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv8d7\" (UniqueName: \"kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.067242 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.067082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.167790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.167759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.167790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.167791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv8d7\" (UniqueName: \"kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.168030 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.167861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.168030 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:21.167988 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-serving-cert: secret "maas-api-e2e-shared-a-a26cce-serving-cert" not found Jul 24 14:19:21.168115 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:21.168059 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls podName:3e66e1f2-8982-4e46-827c-a347ff23292a nodeName:}" failed. No retries permitted until 2026-07-24 14:19:21.66803771 +0000 UTC m=+2455.264601766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls") pod "maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" (UID: "3e66e1f2-8982-4e46-827c-a347ff23292a") : secret "maas-api-e2e-shared-a-a26cce-serving-cert" not found Jul 24 14:19:21.168479 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.168460 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.178578 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.178554 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv8d7\" (UniqueName: \"kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.287545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.287455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:21.506733 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.506701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:21.607798 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.607713 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:21.619938 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.619904 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:19:21.623617 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.623594 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.626040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.626017 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a26cce-dockercfg-9kspk\"" Jul 24 14:19:21.626178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.626017 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a26cce\"" Jul 24 14:19:21.634548 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.634520 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:19:21.673393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.673356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.673575 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.673406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.673575 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.673449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjk9\" (UniqueName: \"kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.675946 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.675915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") pod \"maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.774540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.774496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.774540 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.774540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjk9\" (UniqueName: \"kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.775488 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.775460 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.783489 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.783460 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjk9\" (UniqueName: \"kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9\") pod \"payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:21.878756 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.878671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:21.935936 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:21.935909 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:22.035092 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.035060 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:19:22.075748 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:22.075694 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e66e1f2_8982_4e46_827c_a347ff23292a.slice/crio-c639f60ff358c61913d721bcc7439ac4f7cd38d1ffd32b507cc2a25e34fc5cd4 WatchSource:0}: Error finding container c639f60ff358c61913d721bcc7439ac4f7cd38d1ffd32b507cc2a25e34fc5cd4: Status 404 returned error can't find the container with id c639f60ff358c61913d721bcc7439ac4f7cd38d1ffd32b507cc2a25e34fc5cd4 Jul 24 14:19:22.095948 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.095926 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:19:22.097549 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:22.097528 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935e5ad7_7b2b_48ae_a0ea_972d8ff65144.slice/crio-b0502179463b953f107ad3bbda032441dfcd76f3f9478501baf1858a95218cdd WatchSource:0}: Error finding container b0502179463b953f107ad3bbda032441dfcd76f3f9478501baf1858a95218cdd: Status 404 returned error can't find the container with id b0502179463b953f107ad3bbda032441dfcd76f3f9478501baf1858a95218cdd Jul 24 14:19:22.256359 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.256333 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:19:22.379772 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379733 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.379772 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-772v9\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379875 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379917 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.379988 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380021 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data\") pod \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\" (UID: \"5e213060-abc5-4005-9cb2-82b7cd7b99f4\") " Jul 24 14:19:22.380381 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380079 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:19:22.380445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380379 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:19:22.380711 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380684 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:19:22.380829 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380704 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:19:22.380829 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380739 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.380829 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.380762 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.381340 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.381317 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:19:22.382498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.382474 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9" (OuterVolumeSpecName: "kube-api-access-772v9") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "kube-api-access-772v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:19:22.382636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.382500 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:19:22.382636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.382550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:19:22.382636 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.382569 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e213060-abc5-4005-9cb2-82b7cd7b99f4" (UID: "5e213060-abc5-4005-9cb2-82b7cd7b99f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:19:22.481812 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481776 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.481812 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481805 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-772v9\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-kube-api-access-772v9\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.481812 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481815 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.482040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481824 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.482040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481832 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.482040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481843 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.482040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.481852 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e213060-abc5-4005-9cb2-82b7cd7b99f4-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:19:22.750258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.750227 2581 generic.go:356] "Generic (PLEG): container finished" podID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerID="782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923" exitCode=0 Jul 24 14:19:22.750445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.750279 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" Jul 24 14:19:22.750445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.750314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerDied","Data":"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923"} Jul 24 14:19:22.750445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.750401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr" event={"ID":"5e213060-abc5-4005-9cb2-82b7cd7b99f4","Type":"ContainerDied","Data":"201348067e5237ba980366fa6c3c9723c9a70056789d11d34733f47a6353f932"} Jul 24 14:19:22.750445 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.750425 2581 scope.go:117] "RemoveContainer" containerID="782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923" Jul 24 14:19:22.752248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.752212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" event={"ID":"935e5ad7-7b2b-48ae-a0ea-972d8ff65144","Type":"ContainerStarted","Data":"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8"} Jul 24 14:19:22.752383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.752255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" event={"ID":"935e5ad7-7b2b-48ae-a0ea-972d8ff65144","Type":"ContainerStarted","Data":"b0502179463b953f107ad3bbda032441dfcd76f3f9478501baf1858a95218cdd"} Jul 24 14:19:22.752383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.752315 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:22.753953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.753926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" event={"ID":"3e66e1f2-8982-4e46-827c-a347ff23292a","Type":"ContainerStarted","Data":"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17"} Jul 24 14:19:22.754066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.754016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" event={"ID":"3e66e1f2-8982-4e46-827c-a347ff23292a","Type":"ContainerStarted","Data":"c639f60ff358c61913d721bcc7439ac4f7cd38d1ffd32b507cc2a25e34fc5cd4"} Jul 24 14:19:22.754066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.754049 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:22.760130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.760110 2581 scope.go:117] "RemoveContainer" containerID="28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59" Jul 24 14:19:22.771346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.771316 2581 scope.go:117] "RemoveContainer" containerID="782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923" Jul 24 14:19:22.771770 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:22.771721 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923\": container with ID starting with 782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923 not found: ID does not exist" containerID="782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923" Jul 24 14:19:22.771874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.771761 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923"} err="failed to get container status \"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923\": rpc error: code = NotFound desc = could not find container \"782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923\": container with ID starting with 782f19a7ff05b55d9bf2191090343788380cd25c1d81c7013e3a8adfad446923 not found: ID does not exist" Jul 24 14:19:22.771874 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.771787 2581 scope.go:117] "RemoveContainer" containerID="28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59" Jul 24 14:19:22.772361 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:19:22.772307 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59\": container with ID starting with 28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59 not found: ID does not exist" containerID="28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59" Jul 24 14:19:22.772465 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.772350 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59"} err="failed to get container status \"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59\": rpc error: code = NotFound desc = could not find container \"28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59\": container with ID starting with 28aeb705dd6fd87eaa69b6f59aab08a2f26e7c1db12728eba0e6a02c5c1dda59 not found: ID does not exist" Jul 24 14:19:22.773993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.773717 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" podStartSLOduration=1.773680836 podStartE2EDuration="1.773680836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:22.772150964 +0000 UTC m=+2456.368715020" watchObservedRunningTime="2026-07-24 14:19:22.773680836 +0000 UTC m=+2456.370244894" Jul 24 14:19:22.792311 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.792259 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" podStartSLOduration=2.23115458 podStartE2EDuration="2.792246659s" totalImagesPullingTime="561.092079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:22.791651222 +0000 UTC m=+2456.388215278" watchObservedRunningTime="2026-07-24 14:19:22.792246659 +0000 UTC m=+2456.388810713" Jul 24 14:19:22.807099 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.807060 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:19:22.813810 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.813785 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-517d91-openshift-default-557bd6fc96-92hqr"] Jul 24 14:19:22.917169 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:22.917144 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" path="/var/lib/kubelet/pods/5e213060-abc5-4005-9cb2-82b7cd7b99f4/volumes" Jul 24 14:19:25.262647 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.262612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:25.625836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.625751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:25.929876 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.929840 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:19:25.930367 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.930326 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:25.930493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.930437 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:25.930493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.930464 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:25.930493 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.930472 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:25.930682 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.930624 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:25.935560 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.935534 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:25.938041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.938018 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-cb9aa5-openshift-default-dockercfg-q244n\"" Jul 24 14:19:25.947345 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.947314 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:19:25.960964 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:25.960926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:26.009265 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6szh\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009458 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009612 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009612 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009724 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.009836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.009766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.110870 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.110828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6szh\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.110900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.110927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.110954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111095 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.110977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111315 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111315 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111315 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111333 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111478 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111717 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111690 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.111803 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.111771 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.112165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.112143 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.113726 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.113700 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.113848 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.113782 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.119467 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.119439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.119692 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.119669 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6szh\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh\") pod \"e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.249169 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.249068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:26.412253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.412227 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:19:26.414485 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:26.414456 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09595d85_7b34_4079_8b95_de68dd53e406.slice/crio-37054f5cfb5b5fc7eb089bd52d3e9d81285a0a70ee7337a18173d1326f76eb48 WatchSource:0}: Error finding container 37054f5cfb5b5fc7eb089bd52d3e9d81285a0a70ee7337a18173d1326f76eb48: Status 404 returned error can't find the container with id 37054f5cfb5b5fc7eb089bd52d3e9d81285a0a70ee7337a18173d1326f76eb48 Jul 24 14:19:26.416902 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.416871 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:26.417007 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.416958 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:26.417084 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.416998 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:26.628962 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.628860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:26.781411 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.781376 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerStarted","Data":"f391c1f794c44f03ca7c4248af00bb4bb3e3d3b1255ef8de1db954c330b931c3"} Jul 24 14:19:26.781643 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.781431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerStarted","Data":"37054f5cfb5b5fc7eb089bd52d3e9d81285a0a70ee7337a18173d1326f76eb48"} Jul 24 14:19:26.809840 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:26.809766 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podStartSLOduration=1.809740384 podStartE2EDuration="1.809740384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:26.807422921 +0000 UTC m=+2460.403986976" watchObservedRunningTime="2026-07-24 14:19:26.809740384 +0000 UTC m=+2460.406304440" Jul 24 14:19:27.249991 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:27.249948 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:27.255204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:27.255177 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:27.787090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:27.787058 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:27.788162 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:27.788141 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:28.777222 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:28.777191 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:19:28.796532 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:28.796499 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:28.896844 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:28.896727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:29.013129 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.013098 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:19:29.013633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.013613 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e213060-abc5-4005-9cb2-82b7cd7b99f4" containerName="istio-proxy" Jul 24 14:19:29.018728 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.018689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.021334 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.021300 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-cb9aa5-serving-cert\"" Jul 24 14:19:29.028790 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.027756 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:19:29.075618 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.072019 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:29.141313 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.141273 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsjw\" (UniqueName: \"kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.141513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.141326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.141513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.141360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.242077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.242021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.242406 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.242375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.242545 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.242518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsjw\" (UniqueName: \"kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.242832 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.242805 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.244912 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.244887 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.251679 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.251649 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsjw\" (UniqueName: \"kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw\") pod \"maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.283253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.283016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:29.336424 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.336383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:29.548756 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.548728 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:19:29.550255 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:29.550231 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59002997_7693_4272_957a_ab7e30a220f4.slice/crio-bcba869318271187ad8a0ed2f46cec54ba41aef5f6cf99882ec6383ca9cb122e WatchSource:0}: Error finding container bcba869318271187ad8a0ed2f46cec54ba41aef5f6cf99882ec6383ca9cb122e: Status 404 returned error can't find the container with id bcba869318271187ad8a0ed2f46cec54ba41aef5f6cf99882ec6383ca9cb122e Jul 24 14:19:29.639110 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.639079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:29.735155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.735119 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:29.743328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.743294 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:19:29.748208 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.748183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.752062 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.752029 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-cb9aa5\"" Jul 24 14:19:29.752062 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.752056 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-cb9aa5-dockercfg-tnnzh\"" Jul 24 14:19:29.761373 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.761340 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:19:29.803969 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.803871 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" event={"ID":"59002997-7693-4272-957a-ab7e30a220f4","Type":"ContainerStarted","Data":"bcba869318271187ad8a0ed2f46cec54ba41aef5f6cf99882ec6383ca9cb122e"} Jul 24 14:19:29.849565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.849528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.849772 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.849579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84476\" (UniqueName: \"kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.950389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.950350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.950651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.950526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84476\" (UniqueName: \"kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.951123 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.951105 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:29.959302 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:29.959269 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84476\" (UniqueName: \"kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476\") pod \"payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:30.061020 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.060914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:30.258610 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.258563 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:19:30.260271 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:19:30.260242 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf216c5_470f_47c4_8152_d9b8d63ecc8c.slice/crio-8303b1e1548372b68540058f7c8183206ed34472afa7bb2769378aa30161945b WatchSource:0}: Error finding container 8303b1e1548372b68540058f7c8183206ed34472afa7bb2769378aa30161945b: Status 404 returned error can't find the container with id 8303b1e1548372b68540058f7c8183206ed34472afa7bb2769378aa30161945b Jul 24 14:19:30.811599 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.811536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" event={"ID":"acf216c5-470f-47c4-8152-d9b8d63ecc8c","Type":"ContainerStarted","Data":"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23"} Jul 24 14:19:30.812103 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.811720 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:30.812103 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.811749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" event={"ID":"acf216c5-470f-47c4-8152-d9b8d63ecc8c","Type":"ContainerStarted","Data":"8303b1e1548372b68540058f7c8183206ed34472afa7bb2769378aa30161945b"} Jul 24 14:19:30.813202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.813176 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" event={"ID":"59002997-7693-4272-957a-ab7e30a220f4","Type":"ContainerStarted","Data":"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378"} Jul 24 14:19:30.813348 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.813307 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:30.830215 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.830159 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" podStartSLOduration=1.8301369520000002 podStartE2EDuration="1.830136952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:30.828455959 +0000 UTC m=+2464.425020015" watchObservedRunningTime="2026-07-24 14:19:30.830136952 +0000 UTC m=+2464.426701056" Jul 24 14:19:30.847163 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:30.847099 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" podStartSLOduration=2.478605065 podStartE2EDuration="2.847077714s" totalImagesPullingTime="368.472649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:30.847018261 +0000 UTC m=+2464.443582317" watchObservedRunningTime="2026-07-24 14:19:30.847077714 +0000 UTC m=+2464.443641767" Jul 24 14:19:33.346492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:33.346452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:33.763426 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:33.763391 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:19:35.591813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.591772 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": read tcp 10.134.0.2:52300->10.134.0.91:15021: read: connection reset by peer" start-of-body= Jul 24 14:19:35.592310 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.591844 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": read tcp 10.134.0.2:52300->10.134.0.91:15021: read: connection reset by peer" Jul 24 14:19:35.843386 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.843285 2581 generic.go:356] "Generic (PLEG): container finished" podID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerID="9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9" exitCode=0 Jul 24 14:19:35.843386 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.843361 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerDied","Data":"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9"} Jul 24 14:19:35.843880 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.843862 2581 scope.go:117] "RemoveContainer" containerID="9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9" Jul 24 14:19:35.845170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.845139 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:35.845293 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.845235 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:35.845293 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:35.845288 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:36.823839 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:36.823808 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:19:36.850983 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:36.850945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerStarted","Data":"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d"} Jul 24 14:19:37.259708 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:37.259670 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:37.259993 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:37.259967 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Jul 24 14:19:37.260091 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:37.260025 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Jul 24 14:19:38.259967 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:38.259930 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Jul 24 14:19:38.260356 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:38.259988 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Jul 24 14:19:39.259143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:39.259108 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Jul 24 14:19:39.259329 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:39.259165 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Jul 24 14:19:40.258951 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:40.258916 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Jul 24 14:19:40.259471 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:40.258981 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Jul 24 14:19:41.260039 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:41.259999 2581 patch_prober.go:28] interesting pod/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Jul 24 14:19:41.260443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:41.260074 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Jul 24 14:19:41.820396 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:41.820361 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:19:42.263066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.263032 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:42.263578 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.263409 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:42.264362 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.264340 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:19:42.789415 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.789378 2581 patch_prober.go:28] interesting pod/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Jul 24 14:19:42.789706 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.789444 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Jul 24 14:19:42.891076 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.891041 2581 generic.go:356] "Generic (PLEG): container finished" podID="09595d85-7b34-4079-8b95-de68dd53e406" containerID="f391c1f794c44f03ca7c4248af00bb4bb3e3d3b1255ef8de1db954c330b931c3" exitCode=0 Jul 24 14:19:42.891276 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.891120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerDied","Data":"f391c1f794c44f03ca7c4248af00bb4bb3e3d3b1255ef8de1db954c330b931c3"} Jul 24 14:19:42.891717 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.891698 2581 scope.go:117] "RemoveContainer" containerID="f391c1f794c44f03ca7c4248af00bb4bb3e3d3b1255ef8de1db954c330b931c3" Jul 24 14:19:42.892792 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.892765 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:42.892887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.892831 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:42.892933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.892885 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:19:42.999269 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:42.999207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:43.897922 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:43.897881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerStarted","Data":"e17a50bceac99ab3d20b6387130780ed940843539e75c154d894c900beb38c0d"} Jul 24 14:19:44.249477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:44.249434 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:44.249700 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:44.249678 2581 patch_prober.go:28] interesting pod/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Jul 24 14:19:44.249756 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:44.249725 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Jul 24 14:19:45.249697 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:45.249654 2581 patch_prober.go:28] interesting pod/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Jul 24 14:19:45.250121 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:45.249717 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Jul 24 14:19:46.249673 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:46.249635 2581 patch_prober.go:28] interesting pod/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Jul 24 14:19:46.249863 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:46.249692 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Jul 24 14:19:47.250337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:47.250297 2581 patch_prober.go:28] interesting pod/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Jul 24 14:19:47.250788 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:47.250358 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Jul 24 14:19:48.253732 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:48.253700 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:48.254138 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:48.254053 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:48.254716 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:48.254696 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:19:52.210728 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:52.210673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:19:55.038118 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:19:55.038085 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:20:11.149829 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:20:11.149787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:20:11.215280 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:20:11.215246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:20:13.416596 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:20:13.416552 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:20:20.561999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:20:20.561920 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:21:12.396008 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:21:12.395957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:22:19.926653 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:19.926559 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:22:21.394933 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:21.394897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:22:30.034142 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.034107 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:22:30.037937 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.037919 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.040474 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.040447 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a26cce\"/\"openshift-service-ca.crt\"" Jul 24 14:22:30.041513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.041382 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a26cce\"/\"sub-test-model-a26cce-kserve-self-signed-certs\"" Jul 24 14:22:30.041513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.041468 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a26cce\"/\"default-dockercfg-jtnct\"" Jul 24 14:22:30.041801 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.041643 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a26cce\"/\"kube-root-ca.crt\"" Jul 24 14:22:30.046675 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.046654 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:22:30.066877 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.066833 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:22:30.098110 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098076 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.098336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbhn\" (UniqueName: \"kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.098336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.098336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098271 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.098336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.098518 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.098354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.198989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbhn\" (UniqueName: \"kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199237 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199237 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199237 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199423 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199423 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199457 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199528 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.199674 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.199619 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.201640 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.201611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.201896 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.201873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.206726 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.206697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbhn\" (UniqueName: \"kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn\") pod \"sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.348868 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.348769 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:30.503882 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.503855 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:22:30.506095 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:22:30.506063 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf7cdb1e_d924_46f6_8d36_1209d6318b7c.slice/crio-025c9672cd8bbf0c4744f11353e1f0373c49ed3f7ae67036abc6c4e1c5ee46d9 WatchSource:0}: Error finding container 025c9672cd8bbf0c4744f11353e1f0373c49ed3f7ae67036abc6c4e1c5ee46d9: Status 404 returned error can't find the container with id 025c9672cd8bbf0c4744f11353e1f0373c49ed3f7ae67036abc6c4e1c5ee46d9 Jul 24 14:22:30.761261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.761224 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" event={"ID":"bf7cdb1e-d924-46f6-8d36-1209d6318b7c","Type":"ContainerStarted","Data":"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1"} Jul 24 14:22:30.761446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.761272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" event={"ID":"bf7cdb1e-d924-46f6-8d36-1209d6318b7c","Type":"ContainerStarted","Data":"025c9672cd8bbf0c4744f11353e1f0373c49ed3f7ae67036abc6c4e1c5ee46d9"} Jul 24 14:22:30.778602 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:30.778524 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" podStartSLOduration=0.778502863 podStartE2EDuration="778.502863ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:22:30.778280787 +0000 UTC m=+2644.374844861" watchObservedRunningTime="2026-07-24 14:22:30.778502863 +0000 UTC m=+2644.375066919" Jul 24 14:22:40.349267 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:40.349210 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:40.349713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:40.349290 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:40.362201 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:40.362171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:40.823130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:40.823103 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:22:59.454327 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:22:59.454285 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:23:46.154889 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:23:46.154853 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:23:46.180284 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:23:46.180245 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:23:46.681072 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:23:46.681042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:23:47.259561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:23:47.259501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:24:01.175064 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.175000 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" podUID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" containerName="main" containerID="cri-o://423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1" gracePeriod=45 Jul 24 14:24:01.432937 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.432867 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:24:01.511681 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511638 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.511681 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511677 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511717 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511762 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511826 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwbhn\" (UniqueName: \"kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511878 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs\") pod \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\" (UID: \"bf7cdb1e-d924-46f6-8d36-1209d6318b7c\") " Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511984 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.511998 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:24:01.512137 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.512038 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home" (OuterVolumeSpecName: "home") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:24:01.512520 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.512174 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-home\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:01.512520 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.512195 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tmp-dir\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:01.512520 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.512209 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-model-cache\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:01.514235 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.514209 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:24:01.514235 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.514225 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn" (OuterVolumeSpecName: "kube-api-access-bwbhn") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "kube-api-access-bwbhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:24:01.514422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.514343 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm" (OuterVolumeSpecName: "dshm") pod "bf7cdb1e-d924-46f6-8d36-1209d6318b7c" (UID: "bf7cdb1e-d924-46f6-8d36-1209d6318b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:24:01.613036 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.612983 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwbhn\" (UniqueName: \"kubernetes.io/projected/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-kube-api-access-bwbhn\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:01.613036 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.613028 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-tls-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:01.613036 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:01.613040 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf7cdb1e-d924-46f6-8d36-1209d6318b7c-dshm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:24:02.223226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.223192 2581 generic.go:356] "Generic (PLEG): container finished" podID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" containerID="423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1" exitCode=0 Jul 24 14:24:02.223723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.223255 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" Jul 24 14:24:02.223723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.223272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" event={"ID":"bf7cdb1e-d924-46f6-8d36-1209d6318b7c","Type":"ContainerDied","Data":"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1"} Jul 24 14:24:02.223723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.223314 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68" event={"ID":"bf7cdb1e-d924-46f6-8d36-1209d6318b7c","Type":"ContainerDied","Data":"025c9672cd8bbf0c4744f11353e1f0373c49ed3f7ae67036abc6c4e1c5ee46d9"} Jul 24 14:24:02.223723 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.223331 2581 scope.go:117] "RemoveContainer" containerID="423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1" Jul 24 14:24:02.233089 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.233066 2581 scope.go:117] "RemoveContainer" containerID="423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1" Jul 24 14:24:02.233423 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:24:02.233403 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1\": container with ID starting with 423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1 not found: ID does not exist" containerID="423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1" Jul 24 14:24:02.233505 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.233434 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1"} err="failed to get container status \"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1\": rpc error: code = NotFound desc = could not find container \"423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1\": container with ID starting with 423befb38a8910025fa235f9b9416d26636182bf3dc69fe93bd64749c593dcc1 not found: ID does not exist" Jul 24 14:24:02.245158 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.245115 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:24:02.247183 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.247153 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a26cce/sub-test-model-a26cce-kserve-85cf4dd8ff-nzs68"] Jul 24 14:24:02.917861 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:02.917825 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" path="/var/lib/kubelet/pods/bf7cdb1e-d924-46f6-8d36-1209d6318b7c/volumes" Jul 24 14:24:03.184572 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:03.184468 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:24:30.734851 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:24:30.734813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:14.565889 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:14.565790 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:15.565570 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:15.565530 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:34.783514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:34.783475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:35.158855 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:35.158760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:35.389165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:35.389127 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:38.117651 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.117611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:38.204733 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.204699 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:38.335561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.335519 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:25:38.335960 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.335946 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" containerName="main" Jul 24 14:25:38.336004 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.335962 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" containerName="main" Jul 24 14:25:38.336044 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.336025 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7cdb1e-d924-46f6-8d36-1209d6318b7c" containerName="main" Jul 24 14:25:38.339134 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.339113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.341328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.341304 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-15e86b-serving-cert\"" Jul 24 14:25:38.352019 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.351988 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:25:38.375978 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.375882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:38.493684 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.493649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:38.504629 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.504574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2t8d\" (UniqueName: \"kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.504789 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.504653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.504789 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.504700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.605675 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.605639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2t8d\" (UniqueName: \"kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.605887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.605697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.605887 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.605732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.606328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.606310 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.608280 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.608256 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.613605 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.613567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2t8d\" (UniqueName: \"kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d\") pod \"maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.651346 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.651244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:38.804881 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.804858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:25:38.807076 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:25:38.807043 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c674e7_6575_45a5_991e_13fa73f27954.slice/crio-6eeb567dd92d343cb308006e47278df1c8abca0e6925f2ca93af6fd453b6d112 WatchSource:0}: Error finding container 6eeb567dd92d343cb308006e47278df1c8abca0e6925f2ca93af6fd453b6d112: Status 404 returned error can't find the container with id 6eeb567dd92d343cb308006e47278df1c8abca0e6925f2ca93af6fd453b6d112 Jul 24 14:25:38.808565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.808546 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:25:38.901171 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:38.901141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:39.004767 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:39.004712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:39.715011 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:39.714968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" event={"ID":"44c674e7-6575-45a5-991e-13fa73f27954","Type":"ContainerStarted","Data":"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f"} Jul 24 14:25:39.715416 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:39.715027 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" event={"ID":"44c674e7-6575-45a5-991e-13fa73f27954","Type":"ContainerStarted","Data":"6eeb567dd92d343cb308006e47278df1c8abca0e6925f2ca93af6fd453b6d112"} Jul 24 14:25:39.715416 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:39.715085 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:39.734341 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:39.734285 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" podStartSLOduration=1.297582847 podStartE2EDuration="1.734269455s" totalImagesPullingTime="436.686608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:39.732101664 +0000 UTC m=+2833.328665722" watchObservedRunningTime="2026-07-24 14:25:39.734269455 +0000 UTC m=+2833.330833539" Jul 24 14:25:40.628656 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:40.628622 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:42.708507 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:42.708455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:45.725916 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:45.725887 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:25:53.711630 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:53.711580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:54.064532 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:54.064376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:54.458336 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:54.458298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.190258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.190214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.287274 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.287237 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.386232 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.386198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.631329 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.631244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.899178 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.899090 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:57.997165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:57.997129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:25:58.000438 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.000408 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:25:58.004195 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.004171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.006725 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.006694 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-03c962\"" Jul 24 14:25:58.016602 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.016555 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:25:58.024008 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.023966 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:25:58.027561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.027535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.030034 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.030009 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-03c962-dockercfg-662ql\"" Jul 24 14:25:58.041038 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.040945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:25:58.083052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.083017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8246h\" (UniqueName: \"kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.083253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.083067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.083253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.083144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.083253 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.083223 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzxn\" (UniqueName: \"kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.184432 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.184388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.184626 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.184463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzxn\" (UniqueName: \"kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.184626 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.184578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8246h\" (UniqueName: \"kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.184748 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.184670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.185234 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.185212 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.185565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.185541 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.194620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.194569 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8246h\" (UniqueName: \"kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h\") pod \"payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.194981 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.194601 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzxn\" (UniqueName: \"kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn\") pod \"payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.316287 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.316248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.341692 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.341660 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.496078 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.496038 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:25:58.497856 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:25:58.497829 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb86db3_96ef_445d_917c_6d3bd55a9a50.slice/crio-f38011ff9392fafa18afe3d8c3ef119d5657936d1779b2d305fbe59542c4c023 WatchSource:0}: Error finding container f38011ff9392fafa18afe3d8c3ef119d5657936d1779b2d305fbe59542c4c023: Status 404 returned error can't find the container with id f38011ff9392fafa18afe3d8c3ef119d5657936d1779b2d305fbe59542c4c023 Jul 24 14:25:58.527744 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.527715 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:25:58.537887 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:25:58.537861 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb06a24a_3712_4fde_a1da_b34e7f63bf15.slice/crio-4f5b732128cec491116b34968d92e47a93a618f3a773cfaa9fe76b6933852d99 WatchSource:0}: Error finding container 4f5b732128cec491116b34968d92e47a93a618f3a773cfaa9fe76b6933852d99: Status 404 returned error can't find the container with id 4f5b732128cec491116b34968d92e47a93a618f3a773cfaa9fe76b6933852d99 Jul 24 14:25:58.813260 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.813162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" event={"ID":"cb06a24a-3712-4fde-a1da-b34e7f63bf15","Type":"ContainerStarted","Data":"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f"} Jul 24 14:25:58.813260 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.813224 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" event={"ID":"cb06a24a-3712-4fde-a1da-b34e7f63bf15","Type":"ContainerStarted","Data":"4f5b732128cec491116b34968d92e47a93a618f3a773cfaa9fe76b6933852d99"} Jul 24 14:25:58.813514 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.813266 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:25:58.814836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.814810 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" event={"ID":"6eb86db3-96ef-445d-917c-6d3bd55a9a50","Type":"ContainerStarted","Data":"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117"} Jul 24 14:25:58.814969 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.814862 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:25:58.814969 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.814880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" event={"ID":"6eb86db3-96ef-445d-917c-6d3bd55a9a50","Type":"ContainerStarted","Data":"f38011ff9392fafa18afe3d8c3ef119d5657936d1779b2d305fbe59542c4c023"} Jul 24 14:25:58.831797 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.831745 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" podStartSLOduration=0.831728292 podStartE2EDuration="831.728292ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:58.830433272 +0000 UTC m=+2852.426997326" watchObservedRunningTime="2026-07-24 14:25:58.831728292 +0000 UTC m=+2852.428292347" Jul 24 14:25:58.849640 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:25:58.849555 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" podStartSLOduration=1.84953667 podStartE2EDuration="1.84953667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:58.848713922 +0000 UTC m=+2852.445278031" watchObservedRunningTime="2026-07-24 14:25:58.84953667 +0000 UTC m=+2852.446100725" Jul 24 14:26:01.737284 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:01.737234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:26:09.822511 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:09.822474 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:26:09.822935 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:09.822549 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:26:19.193266 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:19.193175 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:26:19.325282 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:19.325251 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52_cb06a24a-3712-4fde-a1da-b34e7f63bf15/payload-processing/0.log" Jul 24 14:26:25.507077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:25.507033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:26:56.166228 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.166192 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:26:56.314472 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.314431 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52_cb06a24a-3712-4fde-a1da-b34e7f63bf15/payload-processing/0.log" Jul 24 14:26:56.633704 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.633609 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:26:56.949109 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.949077 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:26:56.952873 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.952848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:56.955384 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.955356 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-d95e17-openshift-default-dockercfg-6kz86\"" Jul 24 14:26:56.966735 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.966646 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:26:56.976771 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:56.976433 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:26:57.002952 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.002912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.002955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.002999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003143 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003140 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003366 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.003366 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.003175 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2nq\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104527 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104565 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.104845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.104780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2nq\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.105283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.105176 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.105424 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.105400 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.105496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.105451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.105496 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.105487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.105626 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.105602 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.107130 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.107109 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.107434 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.107416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.113995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.113961 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.114894 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.114877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2nq\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq\") pod \"e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.207532 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.207425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:26:57.277940 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.277900 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:57.431434 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.431406 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:26:57.433044 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:26:57.433020 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36b7e4c_6d3e_4927_adbf_6569f273e89b.slice/crio-3e6a13d1b7fc7058423af6e3fb161d406f59f9ab9bd15bb3c813a45f3347f755 WatchSource:0}: Error finding container 3e6a13d1b7fc7058423af6e3fb161d406f59f9ab9bd15bb3c813a45f3347f755: Status 404 returned error can't find the container with id 3e6a13d1b7fc7058423af6e3fb161d406f59f9ab9bd15bb3c813a45f3347f755 Jul 24 14:26:57.435492 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.435447 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:26:57.435573 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.435549 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:26:57.435659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:57.435641 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:26:58.133857 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:58.133823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerStarted","Data":"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35"} Jul 24 14:26:58.134051 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:58.133873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerStarted","Data":"3e6a13d1b7fc7058423af6e3fb161d406f59f9ab9bd15bb3c813a45f3347f755"} Jul 24 14:26:58.158925 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:58.158873 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podStartSLOduration=2.158853955 podStartE2EDuration="2.158853955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:26:58.158829822 +0000 UTC m=+2911.755393876" watchObservedRunningTime="2026-07-24 14:26:58.158853955 +0000 UTC m=+2911.755418009" Jul 24 14:26:58.278393 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:58.278359 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:58.283533 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:58.283507 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:59.139309 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:59.139284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:26:59.140373 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:26:59.140356 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:00.155413 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:00.155376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:00.258142 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:00.258095 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:00.371289 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:00.371256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:00.463135 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:00.463093 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:00.989813 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:00.989772 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:01.090065 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.090008 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:01.176026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.175991 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:01.179778 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.179757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.182565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.182536 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d95e17-dockercfg-28vdd\"" Jul 24 14:27:01.182754 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.182638 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d95e17\"" Jul 24 14:27:01.188765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.188701 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:01.246895 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.246790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.247186 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.246993 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplgm\" (UniqueName: \"kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.348052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.348009 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.348248 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.348105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zplgm\" (UniqueName: \"kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.348798 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.348780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.356376 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.356345 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplgm\" (UniqueName: \"kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm\") pod \"payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.492441 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.492402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:01.645958 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:01.645932 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:01.648634 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:27:01.648551 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04ce363_5963_4e5e_8022_6195211c68f2.slice/crio-7d464f12b282a6734b81f48d76fd08aa2e178fd84aa32d012fbbe0dfff302f43 WatchSource:0}: Error finding container 7d464f12b282a6734b81f48d76fd08aa2e178fd84aa32d012fbbe0dfff302f43: Status 404 returned error can't find the container with id 7d464f12b282a6734b81f48d76fd08aa2e178fd84aa32d012fbbe0dfff302f43 Jul 24 14:27:02.156677 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:02.156622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" event={"ID":"c04ce363-5963-4e5e-8022-6195211c68f2","Type":"ContainerStarted","Data":"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002"} Jul 24 14:27:02.156886 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:02.156701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" event={"ID":"c04ce363-5963-4e5e-8022-6195211c68f2","Type":"ContainerStarted","Data":"7d464f12b282a6734b81f48d76fd08aa2e178fd84aa32d012fbbe0dfff302f43"} Jul 24 14:27:02.156886 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:02.156728 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:02.179858 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:02.179786 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" podStartSLOduration=1.179760149 podStartE2EDuration="1.179760149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:27:02.175734619 +0000 UTC m=+2915.772298688" watchObservedRunningTime="2026-07-24 14:27:02.179760149 +0000 UTC m=+2915.776324206" Jul 24 14:27:04.723150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:04.723111 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:13.163913 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:13.163878 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:14.141207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.141111 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Jul 24 14:27:14.141207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.141190 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Jul 24 14:27:14.229561 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.229530 2581 generic.go:356] "Generic (PLEG): container finished" podID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerID="7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35" exitCode=0 Jul 24 14:27:14.229984 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.229632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerDied","Data":"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35"} Jul 24 14:27:14.230070 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.230055 2581 scope.go:117] "RemoveContainer" containerID="7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35" Jul 24 14:27:14.231179 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.231144 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:27:14.231246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.231227 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:27:14.231283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:14.231260 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 14:27:15.237899 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:15.237862 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerStarted","Data":"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac"} Jul 24 14:27:15.279296 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:15.279255 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:15.279629 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:15.279296 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Jul 24 14:27:15.279751 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:15.279710 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Jul 24 14:27:16.281886 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:16.281848 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Jul 24 14:27:16.282906 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:16.282471 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Jul 24 14:27:17.280987 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:17.280929 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Jul 24 14:27:17.281335 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:17.281290 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Jul 24 14:27:18.279086 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:18.279039 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Jul 24 14:27:18.279667 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:18.279113 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Jul 24 14:27:19.282459 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:19.282424 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:19.282906 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:19.282775 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:19.283357 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:19.283340 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:26.367052 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.367018 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:26.381264 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.381231 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:26.381515 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.381487 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" podUID="c04ce363-5963-4e5e-8022-6195211c68f2" containerName="payload-pre-processing" containerID="cri-o://c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002" gracePeriod=30 Jul 24 14:27:26.649930 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.649902 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:26.697204 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.697175 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume\") pod \"c04ce363-5963-4e5e-8022-6195211c68f2\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " Jul 24 14:27:26.697366 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.697317 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zplgm\" (UniqueName: \"kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm\") pod \"c04ce363-5963-4e5e-8022-6195211c68f2\" (UID: \"c04ce363-5963-4e5e-8022-6195211c68f2\") " Jul 24 14:27:26.697565 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.697538 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c04ce363-5963-4e5e-8022-6195211c68f2" (UID: "c04ce363-5963-4e5e-8022-6195211c68f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:26.699435 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.699410 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm" (OuterVolumeSpecName: "kube-api-access-zplgm") pod "c04ce363-5963-4e5e-8022-6195211c68f2" (UID: "c04ce363-5963-4e5e-8022-6195211c68f2"). InnerVolumeSpecName "kube-api-access-zplgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:26.798263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.798223 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplgm\" (UniqueName: \"kubernetes.io/projected/c04ce363-5963-4e5e-8022-6195211c68f2-kube-api-access-zplgm\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:26.798263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:26.798254 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04ce363-5963-4e5e-8022-6195211c68f2-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:27.312117 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.312020 2581 generic.go:356] "Generic (PLEG): container finished" podID="c04ce363-5963-4e5e-8022-6195211c68f2" containerID="c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002" exitCode=0 Jul 24 14:27:27.312117 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.312069 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" Jul 24 14:27:27.312117 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.312104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" event={"ID":"c04ce363-5963-4e5e-8022-6195211c68f2","Type":"ContainerDied","Data":"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002"} Jul 24 14:27:27.312457 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.312160 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf" event={"ID":"c04ce363-5963-4e5e-8022-6195211c68f2","Type":"ContainerDied","Data":"7d464f12b282a6734b81f48d76fd08aa2e178fd84aa32d012fbbe0dfff302f43"} Jul 24 14:27:27.312457 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.312182 2581 scope.go:117] "RemoveContainer" containerID="c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002" Jul 24 14:27:27.321071 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.321055 2581 scope.go:117] "RemoveContainer" containerID="c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002" Jul 24 14:27:27.321337 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:27:27.321316 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002\": container with ID starting with c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002 not found: ID does not exist" containerID="c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002" Jul 24 14:27:27.321387 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.321346 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002"} err="failed to get container status \"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002\": rpc error: code = NotFound desc = could not find container \"c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002\": container with ID starting with c2ca2e4ba26ca814781ac8c5d4e6a193ec60c48e3f46230ba6a76fa6bd9cf002 not found: ID does not exist" Jul 24 14:27:27.332085 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.332057 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:27.335885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.335864 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d95e17-56957548d8-m76kf"] Jul 24 14:27:27.773816 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:27.773780 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:28.924957 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:28.924918 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04ce363-5963-4e5e-8022-6195211c68f2" path="/var/lib/kubelet/pods/c04ce363-5963-4e5e-8022-6195211c68f2/volumes" Jul 24 14:27:30.819529 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:30.819489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:31.208951 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:31.208923 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:37.123457 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.123421 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:37.229562 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.229522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:37.423833 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.423806 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:27:37.424029 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.424005 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" containerID="cri-o://b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac" gracePeriod=30 Jul 24 14:27:37.443621 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.443577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:37.699376 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:37.699276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:42.690539 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.690477 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:42.752739 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752707 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.752926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752769 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.752926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.752926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752828 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.752926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm2nq\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.752926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752914 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.752961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753017 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753065 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data\") pod \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\" (UID: \"d36b7e4c-6d3e-4927-adbf-6569f273e89b\") " Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753138 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753154 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:27:42.753202 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753181 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:42.753528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753358 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:27:42.753528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753480 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.753528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753512 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.753528 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753526 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.753765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.753540 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.754155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.754132 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:27:42.755214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.755183 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:27:42.755301 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.755250 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:27:42.755517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.755496 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:42.755517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.755506 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq" (OuterVolumeSpecName: "kube-api-access-hm2nq") pod "d36b7e4c-6d3e-4927-adbf-6569f273e89b" (UID: "d36b7e4c-6d3e-4927-adbf-6569f273e89b"). InnerVolumeSpecName "kube-api-access-hm2nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:42.855022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.854984 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2nq\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-kube-api-access-hm2nq\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.855022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.855014 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.855022 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.855023 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.855304 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.855038 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:42.855304 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:42.855049 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36b7e4c-6d3e-4927-adbf-6569f273e89b-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:43.397389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.397357 2581 generic.go:356] "Generic (PLEG): container finished" podID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerID="b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac" exitCode=0 Jul 24 14:27:43.397577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.397415 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" Jul 24 14:27:43.397577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.397446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerDied","Data":"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac"} Jul 24 14:27:43.397577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.397519 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm" event={"ID":"d36b7e4c-6d3e-4927-adbf-6569f273e89b","Type":"ContainerDied","Data":"3e6a13d1b7fc7058423af6e3fb161d406f59f9ab9bd15bb3c813a45f3347f755"} Jul 24 14:27:43.397577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.397549 2581 scope.go:117] "RemoveContainer" containerID="b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac" Jul 24 14:27:43.407783 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.407757 2581 scope.go:117] "RemoveContainer" containerID="7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35" Jul 24 14:27:43.416311 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.416286 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:27:43.416796 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.416745 2581 scope.go:117] "RemoveContainer" containerID="b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac" Jul 24 14:27:43.417129 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:27:43.417108 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac\": container with ID starting with b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac not found: ID does not exist" containerID="b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac" Jul 24 14:27:43.417214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.417148 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac"} err="failed to get container status \"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac\": rpc error: code = NotFound desc = could not find container \"b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac\": container with ID starting with b6cc01bfc7da53c6060fec7c775076fe2148e7fd158f83fe6bec6803255c07ac not found: ID does not exist" Jul 24 14:27:43.417214 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.417168 2581 scope.go:117] "RemoveContainer" containerID="7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35" Jul 24 14:27:43.417518 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:27:43.417485 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35\": container with ID starting with 7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35 not found: ID does not exist" containerID="7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35" Jul 24 14:27:43.417683 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.417518 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35"} err="failed to get container status \"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35\": rpc error: code = NotFound desc = could not find container \"7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35\": container with ID starting with 7ae73850c6b209fae186731606ebebdc276c2a4c3f8792fe78a3f3be229f3e35 not found: ID does not exist" Jul 24 14:27:43.419851 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:43.419828 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d95e17-openshift-default-557799bc54-m9bwm"] Jul 24 14:27:44.916547 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:44.916517 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" path="/var/lib/kubelet/pods/d36b7e4c-6d3e-4927-adbf-6569f273e89b/volumes" Jul 24 14:27:54.040814 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:54.040724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:54.111152 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:54.111119 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:56.901062 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:56.901027 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:27:56.901564 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:56.901339 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" podUID="44c674e7-6575-45a5-991e-13fa73f27954" containerName="maas-api" containerID="cri-o://04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f" gracePeriod=30 Jul 24 14:27:56.923850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:56.923817 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:57.010899 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.010865 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:57.076534 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.076501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:27:57.173289 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.173262 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:27:57.286156 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.286117 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca\") pod \"44c674e7-6575-45a5-991e-13fa73f27954\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " Jul 24 14:27:57.286337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.286178 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2t8d\" (UniqueName: \"kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d\") pod \"44c674e7-6575-45a5-991e-13fa73f27954\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " Jul 24 14:27:57.286337 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.286274 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls\") pod \"44c674e7-6575-45a5-991e-13fa73f27954\" (UID: \"44c674e7-6575-45a5-991e-13fa73f27954\") " Jul 24 14:27:57.286512 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.286488 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44c674e7-6575-45a5-991e-13fa73f27954" (UID: "44c674e7-6575-45a5-991e-13fa73f27954"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:57.288400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.288367 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d" (OuterVolumeSpecName: "kube-api-access-b2t8d") pod "44c674e7-6575-45a5-991e-13fa73f27954" (UID: "44c674e7-6575-45a5-991e-13fa73f27954"). InnerVolumeSpecName "kube-api-access-b2t8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:57.288400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.288393 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44c674e7-6575-45a5-991e-13fa73f27954" (UID: "44c674e7-6575-45a5-991e-13fa73f27954"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:27:57.387259 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.387220 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44c674e7-6575-45a5-991e-13fa73f27954-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.387259 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.387255 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44c674e7-6575-45a5-991e-13fa73f27954-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.387259 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.387266 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2t8d\" (UniqueName: \"kubernetes.io/projected/44c674e7-6575-45a5-991e-13fa73f27954-kube-api-access-b2t8d\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.473845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.473798 2581 generic.go:356] "Generic (PLEG): container finished" podID="44c674e7-6575-45a5-991e-13fa73f27954" containerID="04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f" exitCode=0 Jul 24 14:27:57.473845 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.473845 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" Jul 24 14:27:57.474097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.473876 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" event={"ID":"44c674e7-6575-45a5-991e-13fa73f27954","Type":"ContainerDied","Data":"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f"} Jul 24 14:27:57.474097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.473931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89" event={"ID":"44c674e7-6575-45a5-991e-13fa73f27954","Type":"ContainerDied","Data":"6eeb567dd92d343cb308006e47278df1c8abca0e6925f2ca93af6fd453b6d112"} Jul 24 14:27:57.474097 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.473956 2581 scope.go:117] "RemoveContainer" containerID="04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f" Jul 24 14:27:57.483569 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.483546 2581 scope.go:117] "RemoveContainer" containerID="04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f" Jul 24 14:27:57.483912 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:27:57.483894 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f\": container with ID starting with 04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f not found: ID does not exist" containerID="04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f" Jul 24 14:27:57.483986 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.483920 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f"} err="failed to get container status \"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f\": rpc error: code = NotFound desc = could not find container \"04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f\": container with ID starting with 04ef77df268d2f31a374cc2d894afb12e3b9199f44111cda5e7ddc7cffe9447f not found: ID does not exist" Jul 24 14:27:57.504390 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.504357 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:27:57.512163 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:57.512130 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-15e86b-dbbd8c898-cpz89"] Jul 24 14:27:58.917513 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:27:58.917466 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c674e7-6575-45a5-991e-13fa73f27954" path="/var/lib/kubelet/pods/44c674e7-6575-45a5-991e-13fa73f27954/volumes" Jul 24 14:28:01.431041 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:01.431007 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:01.841166 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:01.841066 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:06.914311 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:06.914276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:39.301221 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:39.301190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:43.731063 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:43.731031 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:44.386917 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:44.386874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:50.052498 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:50.052459 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:50.153286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:50.153250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:50.335108 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:50.335020 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:50.578215 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:50.578184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:28:58.761384 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:28:58.761351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:01.288853 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.288820 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:01.316056 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.316018 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:29:01.316277 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.316250 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" podUID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" containerName="payload-processing" containerID="cri-o://06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f" gracePeriod=30 Jul 24 14:29:01.324711 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.324675 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:29:01.325154 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.325120 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" podUID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" containerName="payload-pre-processing" containerID="cri-o://64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117" gracePeriod=30 Jul 24 14:29:01.389187 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.389153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:01.469342 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.469294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:01.600410 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.600379 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:29:01.621791 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.621765 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:29:01.663286 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.663253 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8246h\" (UniqueName: \"kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h\") pod \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " Jul 24 14:29:01.663517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.663363 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmzxn\" (UniqueName: \"kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn\") pod \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " Jul 24 14:29:01.663517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.663395 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume\") pod \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\" (UID: \"6eb86db3-96ef-445d-917c-6d3bd55a9a50\") " Jul 24 14:29:01.663517 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.663422 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume\") pod \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\" (UID: \"cb06a24a-3712-4fde-a1da-b34e7f63bf15\") " Jul 24 14:29:01.664155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.664104 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6eb86db3-96ef-445d-917c-6d3bd55a9a50" (UID: "6eb86db3-96ef-445d-917c-6d3bd55a9a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:01.664155 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.664117 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb06a24a-3712-4fde-a1da-b34e7f63bf15" (UID: "cb06a24a-3712-4fde-a1da-b34e7f63bf15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:01.665849 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.665822 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h" (OuterVolumeSpecName: "kube-api-access-8246h") pod "6eb86db3-96ef-445d-917c-6d3bd55a9a50" (UID: "6eb86db3-96ef-445d-917c-6d3bd55a9a50"). InnerVolumeSpecName "kube-api-access-8246h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:01.666081 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.666063 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn" (OuterVolumeSpecName: "kube-api-access-wmzxn") pod "cb06a24a-3712-4fde-a1da-b34e7f63bf15" (UID: "cb06a24a-3712-4fde-a1da-b34e7f63bf15"). InnerVolumeSpecName "kube-api-access-wmzxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:01.764808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.764771 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzxn\" (UniqueName: \"kubernetes.io/projected/cb06a24a-3712-4fde-a1da-b34e7f63bf15-kube-api-access-wmzxn\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.764808 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.764807 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb86db3-96ef-445d-917c-6d3bd55a9a50-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.765014 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.764821 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb06a24a-3712-4fde-a1da-b34e7f63bf15-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.765014 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.764834 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8246h\" (UniqueName: \"kubernetes.io/projected/6eb86db3-96ef-445d-917c-6d3bd55a9a50-kube-api-access-8246h\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.799063 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.799031 2581 generic.go:356] "Generic (PLEG): container finished" podID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" containerID="06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f" exitCode=0 Jul 24 14:29:01.799246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.799076 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" Jul 24 14:29:01.799246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.799120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" event={"ID":"cb06a24a-3712-4fde-a1da-b34e7f63bf15","Type":"ContainerDied","Data":"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f"} Jul 24 14:29:01.799246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.799180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52" event={"ID":"cb06a24a-3712-4fde-a1da-b34e7f63bf15","Type":"ContainerDied","Data":"4f5b732128cec491116b34968d92e47a93a618f3a773cfaa9fe76b6933852d99"} Jul 24 14:29:01.799246 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.799208 2581 scope.go:117] "RemoveContainer" containerID="06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f" Jul 24 14:29:01.800546 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.800520 2581 generic.go:356] "Generic (PLEG): container finished" podID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" containerID="64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117" exitCode=0 Jul 24 14:29:01.800685 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.800569 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" Jul 24 14:29:01.800685 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.800649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" event={"ID":"6eb86db3-96ef-445d-917c-6d3bd55a9a50","Type":"ContainerDied","Data":"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117"} Jul 24 14:29:01.800789 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.800702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc" event={"ID":"6eb86db3-96ef-445d-917c-6d3bd55a9a50","Type":"ContainerDied","Data":"f38011ff9392fafa18afe3d8c3ef119d5657936d1779b2d305fbe59542c4c023"} Jul 24 14:29:01.809959 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.809836 2581 scope.go:117] "RemoveContainer" containerID="06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f" Jul 24 14:29:01.811150 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:01.810392 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f\": container with ID starting with 06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f not found: ID does not exist" containerID="06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f" Jul 24 14:29:01.811150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.810434 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f"} err="failed to get container status \"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f\": rpc error: code = NotFound desc = could not find container \"06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f\": container with ID starting with 06d219dfb4a37250abd2b5502795fdff3209c23afe030ba2511ef7da2715d37f not found: ID does not exist" Jul 24 14:29:01.811150 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.810462 2581 scope.go:117] "RemoveContainer" containerID="64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117" Jul 24 14:29:01.821182 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.821156 2581 scope.go:117] "RemoveContainer" containerID="64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117" Jul 24 14:29:01.821613 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:01.821571 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117\": container with ID starting with 64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117 not found: ID does not exist" containerID="64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117" Jul 24 14:29:01.821696 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.821629 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117"} err="failed to get container status \"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117\": rpc error: code = NotFound desc = could not find container \"64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117\": container with ID starting with 64b6f621d9b89512c8aa94731601e3c58a932807befe12d5a6bb055d29283117 not found: ID does not exist" Jul 24 14:29:01.826703 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.826625 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:29:01.830701 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.830676 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03c962-7775c9dfcd-jfk52"] Jul 24 14:29:01.840604 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.840560 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:29:01.844758 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:01.844726 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03c962-6f747584fb-59rcc"] Jul 24 14:29:02.916380 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:02.916344 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" path="/var/lib/kubelet/pods/6eb86db3-96ef-445d-917c-6d3bd55a9a50/volumes" Jul 24 14:29:02.916784 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:02.916731 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" path="/var/lib/kubelet/pods/cb06a24a-3712-4fde-a1da-b34e7f63bf15/volumes" Jul 24 14:29:05.780702 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:05.780668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:06.176516 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:06.176478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:12.286287 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:12.286251 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:12.447090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:12.447057 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:12.604159 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:12.604061 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:12.866687 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:12.866604 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:23.460114 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.460021 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:29:23.460560 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.460279 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerName="maas-api" containerID="cri-o://c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17" gracePeriod=30 Jul 24 14:29:23.489637 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.489611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:23.516968 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.516933 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:29:23.517249 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.517220 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerName="payload-processing" containerID="cri-o://3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8" gracePeriod=30 Jul 24 14:29:23.589885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.589836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:23.696379 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.696308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:23.772800 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.772772 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:29:23.801877 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.801851 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:29:23.869775 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.869737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") pod \"3e66e1f2-8982-4e46-827c-a347ff23292a\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " Jul 24 14:29:23.869977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.869829 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume\") pod \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " Jul 24 14:29:23.869977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.869884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pjk9\" (UniqueName: \"kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9\") pod \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\" (UID: \"935e5ad7-7b2b-48ae-a0ea-972d8ff65144\") " Jul 24 14:29:23.869977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.869918 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca\") pod \"3e66e1f2-8982-4e46-827c-a347ff23292a\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " Jul 24 14:29:23.869977 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.869941 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv8d7\" (UniqueName: \"kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7\") pod \"3e66e1f2-8982-4e46-827c-a347ff23292a\" (UID: \"3e66e1f2-8982-4e46-827c-a347ff23292a\") " Jul 24 14:29:23.870417 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.870382 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "935e5ad7-7b2b-48ae-a0ea-972d8ff65144" (UID: "935e5ad7-7b2b-48ae-a0ea-972d8ff65144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:23.870544 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.870389 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e66e1f2-8982-4e46-827c-a347ff23292a" (UID: "3e66e1f2-8982-4e46-827c-a347ff23292a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:23.872199 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.872176 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9" (OuterVolumeSpecName: "kube-api-access-9pjk9") pod "935e5ad7-7b2b-48ae-a0ea-972d8ff65144" (UID: "935e5ad7-7b2b-48ae-a0ea-972d8ff65144"). InnerVolumeSpecName "kube-api-access-9pjk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:23.872475 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.872454 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7" (OuterVolumeSpecName: "kube-api-access-kv8d7") pod "3e66e1f2-8982-4e46-827c-a347ff23292a" (UID: "3e66e1f2-8982-4e46-827c-a347ff23292a"). InnerVolumeSpecName "kube-api-access-kv8d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:23.872524 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.872470 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e66e1f2-8982-4e46-827c-a347ff23292a" (UID: "3e66e1f2-8982-4e46-827c-a347ff23292a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:29:23.914149 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.914116 2581 generic.go:356] "Generic (PLEG): container finished" podID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerID="3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8" exitCode=0 Jul 24 14:29:23.914355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.914159 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" Jul 24 14:29:23.914355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.914199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" event={"ID":"935e5ad7-7b2b-48ae-a0ea-972d8ff65144","Type":"ContainerDied","Data":"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8"} Jul 24 14:29:23.914355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.914245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" event={"ID":"935e5ad7-7b2b-48ae-a0ea-972d8ff65144","Type":"ContainerDied","Data":"b0502179463b953f107ad3bbda032441dfcd76f3f9478501baf1858a95218cdd"} Jul 24 14:29:23.914355 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.914271 2581 scope.go:117] "RemoveContainer" containerID="3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8" Jul 24 14:29:23.915705 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.915672 2581 generic.go:356] "Generic (PLEG): container finished" podID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerID="c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17" exitCode=0 Jul 24 14:29:23.915826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.915713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" event={"ID":"3e66e1f2-8982-4e46-827c-a347ff23292a","Type":"ContainerDied","Data":"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17"} Jul 24 14:29:23.915826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.915737 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" Jul 24 14:29:23.915826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.915764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" event={"ID":"3e66e1f2-8982-4e46-827c-a347ff23292a","Type":"ContainerDied","Data":"c639f60ff358c61913d721bcc7439ac4f7cd38d1ffd32b507cc2a25e34fc5cd4"} Jul 24 14:29:23.923296 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.923277 2581 scope.go:117] "RemoveContainer" containerID="3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8" Jul 24 14:29:23.923658 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:23.923629 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8\": container with ID starting with 3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8 not found: ID does not exist" containerID="3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8" Jul 24 14:29:23.923752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.923671 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8"} err="failed to get container status \"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8\": rpc error: code = NotFound desc = could not find container \"3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8\": container with ID starting with 3a4ace14dfd092cab42e4326359379de6cb6f3df74933b6c39930dc11d2a69f8 not found: ID does not exist" Jul 24 14:29:23.923752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.923694 2581 scope.go:117] "RemoveContainer" containerID="c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17" Jul 24 14:29:23.932416 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.932396 2581 scope.go:117] "RemoveContainer" containerID="c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17" Jul 24 14:29:23.932758 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:23.932732 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17\": container with ID starting with c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17 not found: ID does not exist" containerID="c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17" Jul 24 14:29:23.932842 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.932764 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17"} err="failed to get container status \"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17\": rpc error: code = NotFound desc = could not find container \"c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17\": container with ID starting with c1dcebf43afd8f09848aa27475abbb1b4fd5f712347f7d0d4bec70b2afad6a17 not found: ID does not exist" Jul 24 14:29:23.940915 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.940884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:29:23.947853 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.947830 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92"] Jul 24 14:29:23.958633 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.958608 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:29:23.962574 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.962552 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf"] Jul 24 14:29:23.970995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.970968 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.970995 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.970993 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pjk9\" (UniqueName: \"kubernetes.io/projected/935e5ad7-7b2b-48ae-a0ea-972d8ff65144-kube-api-access-9pjk9\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.971145 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.971003 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e66e1f2-8982-4e46-827c-a347ff23292a-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.971145 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.971012 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv8d7\" (UniqueName: \"kubernetes.io/projected/3e66e1f2-8982-4e46-827c-a347ff23292a-kube-api-access-kv8d7\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.971145 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:23.971022 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e66e1f2-8982-4e46-827c-a347ff23292a-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:24.762389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:24.762293 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-a26cce-8955b849d-p9k92" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.93:9004: i/o timeout" Jul 24 14:29:24.916217 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:24.916182 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" path="/var/lib/kubelet/pods/3e66e1f2-8982-4e46-827c-a347ff23292a/volumes" Jul 24 14:29:24.916620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:24.916594 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" path="/var/lib/kubelet/pods/935e5ad7-7b2b-48ae-a0ea-972d8ff65144/volumes" Jul 24 14:29:26.761907 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:26.761866 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a26cce-8cc75d9c7-kdjsf" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.92:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 24 14:29:28.107684 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:28.107643 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:28.514966 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:28.514933 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:35.182971 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.182930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:35.341847 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.341803 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:35.448339 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.442167 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:29:35.448339 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.442465 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" containerID="cri-o://12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d" gracePeriod=30 Jul 24 14:29:35.461880 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.461830 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:35.727480 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:35.727379 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:40.709232 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.709200 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:29:40.810190 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810103 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810190 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810154 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810218 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810249 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810273 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810307 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810340 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810370 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcnd9\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810427 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810399 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo\") pod \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\" (UID: \"0ca6835b-bc1b-4193-bd32-d76e96f7a26f\") " Jul 24 14:29:40.810826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810577 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:29:40.810826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810621 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:29:40.810826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810752 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:40.810826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810785 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:29:40.810826 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810796 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.811101 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.810837 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.811295 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.811266 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:29:40.812499 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.812467 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:40.812638 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.812612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9" (OuterVolumeSpecName: "kube-api-access-qcnd9") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "kube-api-access-qcnd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:40.812728 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.812713 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:29:40.812770 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.812710 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ca6835b-bc1b-4193-bd32-d76e96f7a26f" (UID: "0ca6835b-bc1b-4193-bd32-d76e96f7a26f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:29:40.911449 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911408 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911470 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911484 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnd9\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-kube-api-access-qcnd9\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911505 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911517 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911528 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.911659 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:40.911538 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca6835b-bc1b-4193-bd32-d76e96f7a26f-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:41.001577 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.001541 2581 generic.go:356] "Generic (PLEG): container finished" podID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerID="12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d" exitCode=0 Jul 24 14:29:41.001747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.001625 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" Jul 24 14:29:41.001747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.001632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerDied","Data":"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d"} Jul 24 14:29:41.001747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.001694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj" event={"ID":"0ca6835b-bc1b-4193-bd32-d76e96f7a26f","Type":"ContainerDied","Data":"ee240e338c7068a233d718fed3d4655f3fe08f6bdc06ae6fc954ced35f046318"} Jul 24 14:29:41.001747 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.001722 2581 scope.go:117] "RemoveContainer" containerID="12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d" Jul 24 14:29:41.010491 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.010471 2581 scope.go:117] "RemoveContainer" containerID="9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9" Jul 24 14:29:41.018908 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.018888 2581 scope.go:117] "RemoveContainer" containerID="12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d" Jul 24 14:29:41.019208 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:41.019191 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d\": container with ID starting with 12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d not found: ID does not exist" containerID="12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d" Jul 24 14:29:41.019290 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.019221 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d"} err="failed to get container status \"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d\": rpc error: code = NotFound desc = could not find container \"12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d\": container with ID starting with 12d0df1f02789034010a3c37806dbb19e670cbd06e7a7f75dfc72161f839192d not found: ID does not exist" Jul 24 14:29:41.019290 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.019246 2581 scope.go:117] "RemoveContainer" containerID="9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9" Jul 24 14:29:41.019509 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:41.019482 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9\": container with ID starting with 9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9 not found: ID does not exist" containerID="9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9" Jul 24 14:29:41.019557 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.019517 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9"} err="failed to get container status \"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9\": rpc error: code = NotFound desc = could not find container \"9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9\": container with ID starting with 9cc0a26fd74194132e4052450153ba9057ff1dd2646aea42854026c5f650e5a9 not found: ID does not exist" Jul 24 14:29:41.023035 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.023011 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:29:41.031960 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:41.031930 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a26cce-openshift-default-75dd66c69b-xpjwj"] Jul 24 14:29:42.916439 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:42.916405 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" path="/var/lib/kubelet/pods/0ca6835b-bc1b-4193-bd32-d76e96f7a26f/volumes" Jul 24 14:29:46.298765 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.298732 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:29:46.299349 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.298991 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" podUID="59002997-7693-4272-957a-ab7e30a220f4" containerName="maas-api" containerID="cri-o://43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378" gracePeriod=30 Jul 24 14:29:46.329125 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.329094 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:46.376853 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.376809 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:29:46.377185 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.377135 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" podUID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" containerName="payload-pre-processing" containerID="cri-o://3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23" gracePeriod=30 Jul 24 14:29:46.430372 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.430329 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:46.530847 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.530797 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:46.635978 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.635948 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:29:46.670077 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.670051 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:29:46.763729 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.763687 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca\") pod \"59002997-7693-4272-957a-ab7e30a220f4\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " Jul 24 14:29:46.763939 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.763779 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls\") pod \"59002997-7693-4272-957a-ab7e30a220f4\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " Jul 24 14:29:46.763939 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.763838 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsjw\" (UniqueName: \"kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw\") pod \"59002997-7693-4272-957a-ab7e30a220f4\" (UID: \"59002997-7693-4272-957a-ab7e30a220f4\") " Jul 24 14:29:46.764176 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.764150 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59002997-7693-4272-957a-ab7e30a220f4" (UID: "59002997-7693-4272-957a-ab7e30a220f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:46.766102 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.766069 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw" (OuterVolumeSpecName: "kube-api-access-zhsjw") pod "59002997-7693-4272-957a-ab7e30a220f4" (UID: "59002997-7693-4272-957a-ab7e30a220f4"). InnerVolumeSpecName "kube-api-access-zhsjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:46.766459 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.766432 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59002997-7693-4272-957a-ab7e30a220f4" (UID: "59002997-7693-4272-957a-ab7e30a220f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:29:46.864831 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.864731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume\") pod \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " Jul 24 14:29:46.864831 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.864789 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84476\" (UniqueName: \"kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476\") pod \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\" (UID: \"acf216c5-470f-47c4-8152-d9b8d63ecc8c\") " Jul 24 14:29:46.865053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.864989 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59002997-7693-4272-957a-ab7e30a220f4-maas-api-tls\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:46.865053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.865007 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsjw\" (UniqueName: \"kubernetes.io/projected/59002997-7693-4272-957a-ab7e30a220f4-kube-api-access-zhsjw\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:46.865053 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.865022 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59002997-7693-4272-957a-ab7e30a220f4-openshift-service-ca\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:46.865239 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.865214 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acf216c5-470f-47c4-8152-d9b8d63ecc8c" (UID: "acf216c5-470f-47c4-8152-d9b8d63ecc8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:46.867088 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.867052 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476" (OuterVolumeSpecName: "kube-api-access-84476") pod "acf216c5-470f-47c4-8152-d9b8d63ecc8c" (UID: "acf216c5-470f-47c4-8152-d9b8d63ecc8c"). InnerVolumeSpecName "kube-api-access-84476". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:46.966660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.966621 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84476\" (UniqueName: \"kubernetes.io/projected/acf216c5-470f-47c4-8152-d9b8d63ecc8c-kube-api-access-84476\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:46.966660 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:46.966661 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf216c5-470f-47c4-8152-d9b8d63ecc8c-plugins-config-volume\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:29:47.033064 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.033030 2581 generic.go:356] "Generic (PLEG): container finished" podID="59002997-7693-4272-957a-ab7e30a220f4" containerID="43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378" exitCode=0 Jul 24 14:29:47.033255 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.033072 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" Jul 24 14:29:47.033255 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.033111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" event={"ID":"59002997-7693-4272-957a-ab7e30a220f4","Type":"ContainerDied","Data":"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378"} Jul 24 14:29:47.033255 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.033161 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg" event={"ID":"59002997-7693-4272-957a-ab7e30a220f4","Type":"ContainerDied","Data":"bcba869318271187ad8a0ed2f46cec54ba41aef5f6cf99882ec6383ca9cb122e"} Jul 24 14:29:47.033255 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.033185 2581 scope.go:117] "RemoveContainer" containerID="43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378" Jul 24 14:29:47.034477 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.034455 2581 generic.go:356] "Generic (PLEG): container finished" podID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" containerID="3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23" exitCode=0 Jul 24 14:29:47.034613 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.034481 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" event={"ID":"acf216c5-470f-47c4-8152-d9b8d63ecc8c","Type":"ContainerDied","Data":"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23"} Jul 24 14:29:47.034613 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.034503 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" Jul 24 14:29:47.034613 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.034533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55" event={"ID":"acf216c5-470f-47c4-8152-d9b8d63ecc8c","Type":"ContainerDied","Data":"8303b1e1548372b68540058f7c8183206ed34472afa7bb2769378aa30161945b"} Jul 24 14:29:47.042774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.042753 2581 scope.go:117] "RemoveContainer" containerID="43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378" Jul 24 14:29:47.043110 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:47.043085 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378\": container with ID starting with 43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378 not found: ID does not exist" containerID="43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378" Jul 24 14:29:47.043196 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.043120 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378"} err="failed to get container status \"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378\": rpc error: code = NotFound desc = could not find container \"43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378\": container with ID starting with 43e3036ef0acf3756ddee11cff032f40bb60838e1ba9053def33fdcfa321c378 not found: ID does not exist" Jul 24 14:29:47.043196 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.043145 2581 scope.go:117] "RemoveContainer" containerID="3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23" Jul 24 14:29:47.051699 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.051679 2581 scope.go:117] "RemoveContainer" containerID="3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23" Jul 24 14:29:47.052014 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:29:47.051991 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23\": container with ID starting with 3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23 not found: ID does not exist" containerID="3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23" Jul 24 14:29:47.052059 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.052016 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23"} err="failed to get container status \"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23\": rpc error: code = NotFound desc = could not find container \"3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23\": container with ID starting with 3b50be802649fc18c6c67a8fd6da8ef0d04e8c645565b485f1ce1ec9bc871f23 not found: ID does not exist" Jul 24 14:29:47.056431 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.056404 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:29:47.059898 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.059871 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cb9aa5-79d6896599-2chvg"] Jul 24 14:29:47.073005 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.072975 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:29:47.077047 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:47.077022 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-cb9aa5-57579c9cc6-bww55"] Jul 24 14:29:48.916893 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:48.916854 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59002997-7693-4272-957a-ab7e30a220f4" path="/var/lib/kubelet/pods/59002997-7693-4272-957a-ab7e30a220f4/volumes" Jul 24 14:29:48.917283 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:48.917210 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" path="/var/lib/kubelet/pods/acf216c5-470f-47c4-8152-d9b8d63ecc8c/volumes" Jul 24 14:29:50.965209 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:50.965178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:51.353200 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:51.353113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:57.665165 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:57.665129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:57.780446 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:57.780413 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:57.948512 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:57.948412 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:29:57.949094 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:57.949059 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" containerID="cri-o://e17a50bceac99ab3d20b6387130780ed940843539e75c154d894c900beb38c0d" gracePeriod=30 Jul 24 14:29:57.966328 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:57.966299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:29:58.220552 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:29:58.220465 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:30:00.157057 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157020 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685"] Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157368 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157379 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157390 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157395 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157404 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59002997-7693-4272-957a-ab7e30a220f4" containerName="maas-api" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157409 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="59002997-7693-4272-957a-ab7e30a220f4" containerName="maas-api" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157417 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" containerName="payload-pre-processing" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157422 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" containerName="payload-pre-processing" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157429 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c674e7-6575-45a5-991e-13fa73f27954" containerName="maas-api" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157434 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c674e7-6575-45a5-991e-13fa73f27954" containerName="maas-api" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157441 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" containerName="payload-pre-processing" Jul 24 14:30:00.157443 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157446 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157454 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" containerName="payload-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157459 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" containerName="payload-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157470 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerName="payload-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157475 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerName="payload-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157483 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerName="maas-api" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157488 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerName="maas-api" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157497 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04ce363-5963-4e5e-8022-6195211c68f2" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157502 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04ce363-5963-4e5e-8022-6195211c68f2" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157550 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e66e1f2-8982-4e46-827c-a347ff23292a" containerName="maas-api" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157556 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c674e7-6575-45a5-991e-13fa73f27954" containerName="maas-api" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157561 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157567 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="59002997-7693-4272-957a-ab7e30a220f4" containerName="maas-api" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157573 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf216c5-470f-47c4-8152-d9b8d63ecc8c" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157579 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb06a24a-3712-4fde-a1da-b34e7f63bf15" containerName="payload-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157602 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157609 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb86db3-96ef-445d-917c-6d3bd55a9a50" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157619 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04ce363-5963-4e5e-8022-6195211c68f2" containerName="payload-pre-processing" Jul 24 14:30:00.157850 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.157626 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="935e5ad7-7b2b-48ae-a0ea-972d8ff65144" containerName="payload-processing" Jul 24 14:30:00.162170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.162149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" Jul 24 14:30:00.166166 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.166140 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:30:00.172349 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.172327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" Jul 24 14:30:00.174294 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.174268 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685"] Jul 24 14:30:00.321110 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:00.321084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685"] Jul 24 14:30:00.322165 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:30:00.322140 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b133ac_d800_43af_84e0_000f13175e27.slice/crio-d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620 WatchSource:0}: Error finding container d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620: Status 404 returned error can't find the container with id d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620 Jul 24 14:30:01.104332 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:01.104234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" event={"ID":"d4b133ac-d800-43af-84e0-000f13175e27","Type":"ContainerStarted","Data":"f074816459ef9a1bfb4c0b8618ef566fd6a89955382feff404c82144d64b71b4"} Jul 24 14:30:01.104332 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:01.104288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" event={"ID":"d4b133ac-d800-43af-84e0-000f13175e27","Type":"ContainerStarted","Data":"d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620"} Jul 24 14:30:01.124096 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:01.124042 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" podStartSLOduration=1.124022712 podStartE2EDuration="1.124022712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:30:01.12212286 +0000 UTC m=+3094.718686917" watchObservedRunningTime="2026-07-24 14:30:01.124022712 +0000 UTC m=+3094.720586767" Jul 24 14:30:02.109999 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:02.109963 2581 generic.go:356] "Generic (PLEG): container finished" podID="d4b133ac-d800-43af-84e0-000f13175e27" containerID="f074816459ef9a1bfb4c0b8618ef566fd6a89955382feff404c82144d64b71b4" exitCode=0 Jul 24 14:30:02.110400 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:02.110052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" event={"ID":"d4b133ac-d800-43af-84e0-000f13175e27","Type":"ContainerDied","Data":"f074816459ef9a1bfb4c0b8618ef566fd6a89955382feff404c82144d64b71b4"} Jul 24 14:30:03.116546 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.116515 2581 generic.go:356] "Generic (PLEG): container finished" podID="09595d85-7b34-4079-8b95-de68dd53e406" containerID="e17a50bceac99ab3d20b6387130780ed940843539e75c154d894c900beb38c0d" exitCode=0 Jul 24 14:30:03.116976 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.116575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerDied","Data":"e17a50bceac99ab3d20b6387130780ed940843539e75c154d894c900beb38c0d"} Jul 24 14:30:03.116976 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.116657 2581 scope.go:117] "RemoveContainer" containerID="f391c1f794c44f03ca7c4248af00bb4bb3e3d3b1255ef8de1db954c330b931c3" Jul 24 14:30:03.222823 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.222748 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:30:03.269675 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.269647 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" Jul 24 14:30:03.313050 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313010 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313050 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313048 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313302 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313084 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313302 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313132 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6szh\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313302 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313152 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313302 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313195 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313490 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313365 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313490 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313394 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313490 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token\") pod \"09595d85-7b34-4079-8b95-de68dd53e406\" (UID: \"09595d85-7b34-4079-8b95-de68dd53e406\") " Jul 24 14:30:03.313662 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313557 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:30:03.313725 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313665 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:30:03.313849 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313831 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-certs\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.313908 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313856 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-workload-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.313908 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.313889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:30:03.314040 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.314020 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:30:03.314460 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.314423 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data" (OuterVolumeSpecName: "istio-data") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:30:03.315634 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.315606 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 14:30:03.315752 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.315639 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token" (OuterVolumeSpecName: "istio-token") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:30:03.315910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.315889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh" (OuterVolumeSpecName: "kube-api-access-m6szh") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "kube-api-access-m6szh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:30:03.315910 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.315897 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09595d85-7b34-4079-8b95-de68dd53e406" (UID: "09595d85-7b34-4079-8b95-de68dd53e406"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 14:30:03.414885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.414841 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09595d85-7b34-4079-8b95-de68dd53e406-istio-podinfo\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415132 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-credential-socket\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415170 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415162 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-istio-token\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415182 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-data\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415194 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09595d85-7b34-4079-8b95-de68dd53e406-istiod-ca-cert\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415206 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09595d85-7b34-4079-8b95-de68dd53e406-istio-envoy\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:03.415350 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:03.415219 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6szh\" (UniqueName: \"kubernetes.io/projected/09595d85-7b34-4079-8b95-de68dd53e406-kube-api-access-m6szh\") on node \"ip-10-0-143-193.ec2.internal\" DevicePath \"\"" Jul 24 14:30:04.122806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.122772 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" Jul 24 14:30:04.122806 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.122782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748390-qs685" event={"ID":"d4b133ac-d800-43af-84e0-000f13175e27","Type":"ContainerDied","Data":"d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620"} Jul 24 14:30:04.123272 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.122837 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57529327643c113e429d5d36b4d95c64edc2e1b23b9541de17bf7dc13b67620" Jul 24 14:30:04.124063 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.124032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" event={"ID":"09595d85-7b34-4079-8b95-de68dd53e406","Type":"ContainerDied","Data":"37054f5cfb5b5fc7eb089bd52d3e9d81285a0a70ee7337a18173d1326f76eb48"} Jul 24 14:30:04.124063 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.124068 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v" Jul 24 14:30:04.124261 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.124090 2581 scope.go:117] "RemoveContainer" containerID="e17a50bceac99ab3d20b6387130780ed940843539e75c154d894c900beb38c0d" Jul 24 14:30:04.148857 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.148821 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:30:04.153453 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.153420 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-cb9aa5-openshift-default-7d9458dd96-nx49v"] Jul 24 14:30:04.667934 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.667905 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4hhd8_cbc031e5-28d9-4b1b-876e-ab1bd4e7b946/manager/0.log" Jul 24 14:30:04.800308 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.800273 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h2jsc_6f1a1723-1ce9-486e-8e01-e75093a98870/manager/0.log" Jul 24 14:30:04.916558 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.916523 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09595d85-7b34-4079-8b95-de68dd53e406" path="/var/lib/kubelet/pods/09595d85-7b34-4079-8b95-de68dd53e406/volumes" Jul 24 14:30:04.946163 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:04.946077 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b69666bc-7wcrr_ce567867-5598-469e-b323-85cd68ff2e53/manager/0.log" Jul 24 14:30:05.242512 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:05.242433 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9c5tp_8c1e1f82-e037-40d2-92a8-cefff59ae3e1/manager/0.log" Jul 24 14:30:05.669620 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:05.669570 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-vw6hm_cf984aa7-a948-4497-8f92-7abb99aaf835/manager/0.log" Jul 24 14:30:05.678391 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:05.678366 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-vw6hm_cf984aa7-a948-4497-8f92-7abb99aaf835/copy-manifests/0.log" Jul 24 14:30:06.238026 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:06.237990 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748360-bj5xf_7556b2f0-ee9d-428a-96a1-027fcf18e8a3/cleanup/0.log" Jul 24 14:30:06.368984 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:06.368959 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748375-pb858_8ac49f24-b184-4f72-b5ad-aa4bf12a4d46/cleanup/0.log" Jul 24 14:30:06.495069 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:06.494988 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748390-qs685_d4b133ac-d800-43af-84e0-000f13175e27/cleanup/0.log" Jul 24 14:30:06.616953 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:06.616926 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hfn5f_32aefa77-9065-449a-a169-61438c1284e8/postgres/0.log" Jul 24 14:30:08.174111 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.174087 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-4lffm_f39c92bf-6319-4b13-ab01-c06aba5a8fdc/manager/0.log" Jul 24 14:30:08.300191 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.300159 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ddv8w_972d23cb-5bcc-448a-b6ce-cfcc1aee743f/manager/0.log" Jul 24 14:30:08.422085 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.422054 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5ntqw_f6ac7b88-87b5-426e-b0cd-13eb84e239a0/kuadrant-console-plugin/0.log" Jul 24 14:30:08.543563 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.543470 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-c7vzn_894b201b-d82e-48be-9c3d-f46648c37034/registry-server/0.log" Jul 24 14:30:08.669559 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.669529 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8s4jj_bba6fca0-967f-4396-9425-85d96d440879/manager/0.log" Jul 24 14:30:08.918044 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:08.918013 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jn88c_4c52dabe-b5cc-4328-b714-9027e4bac2e0/manager/0.log" Jul 24 14:30:09.553225 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:09.553194 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8885cbd9-fmfxg_809238b9-66fe-4c31-9763-f1bc2aa459f1/kube-auth-proxy/0.log" Jul 24 14:30:09.952705 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:09.952675 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hhkjn_ebd03052-7832-4502-afd8-41270a2f3a39/payload-pre-processing/0.log" Jul 24 14:30:10.083774 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:10.083745 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:30:10.573260 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:10.573233 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-c4mpv_9df8d823-0b4a-4fb0-af0a-756b25622b71/main/0.log" Jul 24 14:30:10.827909 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:10.827826 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4k54x_fabaf22d-f85d-433b-8b87-3c2d701d85b7/main/0.log" Jul 24 14:30:17.026919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.026890 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-njvgn/must-gather-c55w8"] Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027232 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027242 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027256 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027261 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027269 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b133ac-d800-43af-84e0-000f13175e27" containerName="cleanup" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027274 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b133ac-d800-43af-84e0-000f13175e27" containerName="cleanup" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027280 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027286 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027294 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:17.027318 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027298 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:17.027689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027353 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36b7e4c-6d3e-4927-adbf-6569f273e89b" containerName="istio-proxy" Jul 24 14:30:17.027689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027361 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b133ac-d800-43af-84e0-000f13175e27" containerName="cleanup" Jul 24 14:30:17.027689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027368 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca6835b-bc1b-4193-bd32-d76e96f7a26f" containerName="istio-proxy" Jul 24 14:30:17.027689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027375 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.027689 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.027382 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09595d85-7b34-4079-8b95-de68dd53e406" containerName="istio-proxy" Jul 24 14:30:17.030444 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.030427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.034479 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:30:17.034429 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-c55w8\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-njvgn\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" podUID="e17cc304-23b0-4aa3-9eb1-d7fea35e2b52" pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.034690 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:30:17.034517 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-njvgn\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-njvgn\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 24 14:30:17.034690 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:30:17.034569 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-njvgn\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-njvgn\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 24 14:30:17.034690 ip-10-0-143-193 kubenswrapper[2581]: E0724 14:30:17.034570 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-bdtjz\" is forbidden: User \"system:node:ip-10-0-143-193.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-must-gather-njvgn\": no relationship found between node 'ip-10-0-143-193.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-njvgn\"/\"default-dockercfg-bdtjz\"" type="*v1.Secret" Jul 24 14:30:17.056606 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.054530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/must-gather-c55w8"] Jul 24 14:30:17.062154 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.062107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:30:17.130855 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.130813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-must-gather-output\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.131066 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.130871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjtx\" (UniqueName: \"kubernetes.io/projected/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-kube-api-access-hqjtx\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.192836 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.192793 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" limit=1024 Jul 24 14:30:17.231936 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.231890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-must-gather-output\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.232094 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.231955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjtx\" (UniqueName: \"kubernetes.io/projected/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-kube-api-access-hqjtx\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:17.232254 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:17.232234 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-must-gather-output\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:18.204783 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.204740 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njvgn\"/\"kube-root-ca.crt\"" Jul 24 14:30:18.610384 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.610294 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njvgn\"/\"openshift-service-ca.crt\"" Jul 24 14:30:18.613617 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.613573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjtx\" (UniqueName: \"kubernetes.io/projected/e17cc304-23b0-4aa3-9eb1-d7fea35e2b52-kube-api-access-hqjtx\") pod \"must-gather-c55w8\" (UID: \"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52\") " pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:18.629810 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.629783 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-njvgn\"/\"default-dockercfg-bdtjz\"" Jul 24 14:30:18.839919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.839874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/must-gather-c55w8" Jul 24 14:30:18.980383 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:18.980359 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/must-gather-c55w8"] Jul 24 14:30:18.982983 ip-10-0-143-193 kubenswrapper[2581]: W0724 14:30:18.982953 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode17cc304_23b0_4aa3_9eb1_d7fea35e2b52.slice/crio-fb80350a224c71b7a91d4d7844809a74fd75e3e827bef353261a2708552abda2 WatchSource:0}: Error finding container fb80350a224c71b7a91d4d7844809a74fd75e3e827bef353261a2708552abda2: Status 404 returned error can't find the container with id fb80350a224c71b7a91d4d7844809a74fd75e3e827bef353261a2708552abda2 Jul 24 14:30:19.197924 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:19.197894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/must-gather-c55w8" event={"ID":"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52","Type":"ContainerStarted","Data":"fb80350a224c71b7a91d4d7844809a74fd75e3e827bef353261a2708552abda2"} Jul 24 14:30:20.204389 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:20.204360 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/must-gather-c55w8" event={"ID":"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52","Type":"ContainerStarted","Data":"537d0abfa286b29603b7a89dd2076a250e0bbe0531e0d4b07ac9f08f59340533"} Jul 24 14:30:21.211281 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:21.211248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/must-gather-c55w8" event={"ID":"e17cc304-23b0-4aa3-9eb1-d7fea35e2b52","Type":"ContainerStarted","Data":"0e3ce4ac56c9ff8bd0d377ba7df225a098c07b486ccff5ded35697f3b9e189f2"} Jul 24 14:30:21.227226 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:21.227166 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-njvgn/must-gather-c55w8" podStartSLOduration=3.103052932 podStartE2EDuration="4.227143672s" totalImagesPullingTime="1.12409074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:30:21.226079137 +0000 UTC m=+3114.822643192" watchObservedRunningTime="2026-07-24 14:30:21.227143672 +0000 UTC m=+3114.823707728" Jul 24 14:30:21.770397 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:21.770368 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n7ctg_c0b8aee6-bb35-4acf-bcf6-dd596bb799ca/global-pull-secret-syncer/0.log" Jul 24 14:30:21.874714 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:21.874683 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ppq5z_e981bc35-bb8b-4ab2-81a6-c0dfb1ff7d05/konnectivity-agent/0.log" Jul 24 14:30:22.018911 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:22.018879 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-193.ec2.internal_9730198ade56c40795706b2e4638cf6a/haproxy/0.log" Jul 24 14:30:27.428258 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:27.428224 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-4lffm_f39c92bf-6319-4b13-ab01-c06aba5a8fdc/manager/0.log" Jul 24 14:30:27.484136 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:27.484074 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ddv8w_972d23cb-5bcc-448a-b6ce-cfcc1aee743f/manager/0.log" Jul 24 14:30:27.518885 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:27.518857 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5ntqw_f6ac7b88-87b5-426e-b0cd-13eb84e239a0/kuadrant-console-plugin/0.log" Jul 24 14:30:27.577090 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:27.576898 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-c7vzn_894b201b-d82e-48be-9c3d-f46648c37034/registry-server/0.log" Jul 24 14:30:27.806680 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:27.805379 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8s4jj_bba6fca0-967f-4396-9425-85d96d440879/manager/0.log" Jul 24 14:30:28.008817 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:28.008767 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-jn88c_4c52dabe-b5cc-4328-b714-9027e4bac2e0/manager/0.log" Jul 24 14:30:30.255680 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:30.255608 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vrwt7_0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51/node-exporter/0.log" Jul 24 14:30:30.301431 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:30.301394 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vrwt7_0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51/kube-rbac-proxy/0.log" Jul 24 14:30:30.337173 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:30.337113 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vrwt7_0dea3f02-6d7c-4d4d-9c8b-9cc9c6282f51/init-textfile/0.log" Jul 24 14:30:32.549189 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.549151 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn"] Jul 24 14:30:32.555551 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.555519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.561990 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.561956 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn"] Jul 24 14:30:32.698713 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.698673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-podres\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.698926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.698730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-lib-modules\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.698926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.698760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvx4\" (UniqueName: \"kubernetes.io/projected/21550407-3bf9-4cba-8e5c-32576e41a880-kube-api-access-9kvx4\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.698926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.698795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-sys\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.698926 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.698857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-proc\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-lib-modules\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800172 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvx4\" (UniqueName: \"kubernetes.io/projected/21550407-3bf9-4cba-8e5c-32576e41a880-kube-api-access-9kvx4\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800415 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-sys\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800415 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-proc\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800415 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-podres\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800604 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800491 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-podres\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.800668 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.800622 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-lib-modules\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.801044 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.801015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-sys\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.801141 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.801088 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21550407-3bf9-4cba-8e5c-32576e41a880-proc\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.809504 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.809464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvx4\" (UniqueName: \"kubernetes.io/projected/21550407-3bf9-4cba-8e5c-32576e41a880-kube-api-access-9kvx4\") pod \"perf-node-gather-daemonset-7rqtn\" (UID: \"21550407-3bf9-4cba-8e5c-32576e41a880\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:32.869392 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:32.869347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:33.079422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:33.078432 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn"] Jul 24 14:30:33.324730 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:33.324694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" event={"ID":"21550407-3bf9-4cba-8e5c-32576e41a880","Type":"ContainerStarted","Data":"29d45a79ed515bc7984a424efc967d9fe7326af45c7302579cf2496904011817"} Jul 24 14:30:33.324944 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:33.324757 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" event={"ID":"21550407-3bf9-4cba-8e5c-32576e41a880","Type":"ContainerStarted","Data":"74698424f95e91f011df07cf0a7d465fe045d49e6eb5c69311f5ba2a41f6b31b"} Jul 24 14:30:33.324944 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:33.324780 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:33.353098 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:33.352910 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" podStartSLOduration=1.352890903 podStartE2EDuration="1.352890903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:30:33.352212799 +0000 UTC m=+3126.948776855" watchObservedRunningTime="2026-07-24 14:30:33.352890903 +0000 UTC m=+3126.949454959" Jul 24 14:30:35.416272 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:35.416231 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvskz_9eda3025-92fd-4593-91b2-9202cf74299d/dns/0.log" Jul 24 14:30:35.440859 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:35.440826 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvskz_9eda3025-92fd-4593-91b2-9202cf74299d/kube-rbac-proxy/0.log" Jul 24 14:30:35.469605 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:35.469555 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7jz2l_c7b4c7f5-d393-4fc6-9d7a-ef50e39e5294/dns-node-resolver/0.log" Jul 24 14:30:36.187005 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:36.186979 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkjz2_821b48f2-7019-4f72-874d-b0435ba169bc/node-ca/0.log" Jul 24 14:30:37.401907 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:37.401868 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8885cbd9-fmfxg_809238b9-66fe-4c31-9763-f1bc2aa459f1/kube-auth-proxy/0.log" Jul 24 14:30:37.535753 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:37.535714 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hhkjn_ebd03052-7832-4502-afd8-41270a2f3a39/payload-pre-processing/0.log" Jul 24 14:30:37.572304 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:37.572271 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:30:38.459854 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:38.459826 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkzqd_fac8536e-3aa4-4112-aeff-5f6b1079ead3/serve-healthcheck-canary/0.log" Jul 24 14:30:39.108263 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:39.108234 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqkcf_3834f8c9-b9c9-4128-bd5e-cfd2a694e56e/kube-rbac-proxy/0.log" Jul 24 14:30:39.157149 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:39.157123 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqkcf_3834f8c9-b9c9-4128-bd5e-cfd2a694e56e/exporter/0.log" Jul 24 14:30:39.196627 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:39.196598 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cqkcf_3834f8c9-b9c9-4128-bd5e-cfd2a694e56e/extractor/0.log" Jul 24 14:30:39.340422 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:39.340386 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-7rqtn" Jul 24 14:30:42.012889 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.012862 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4hhd8_cbc031e5-28d9-4b1b-876e-ab1bd4e7b946/manager/0.log" Jul 24 14:30:42.263523 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.263426 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h2jsc_6f1a1723-1ce9-486e-8e01-e75093a98870/manager/0.log" Jul 24 14:30:42.437327 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.437280 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b69666bc-7wcrr_ce567867-5598-469e-b323-85cd68ff2e53/manager/0.log" Jul 24 14:30:42.595539 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.595435 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9c5tp_8c1e1f82-e037-40d2-92a8-cefff59ae3e1/manager/0.log" Jul 24 14:30:42.853963 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.853856 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-vw6hm_cf984aa7-a948-4497-8f92-7abb99aaf835/manager/0.log" Jul 24 14:30:42.901048 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:42.901018 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-vw6hm_cf984aa7-a948-4497-8f92-7abb99aaf835/copy-manifests/0.log" Jul 24 14:30:43.048687 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:43.048652 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8885cbd9-fmfxg_809238b9-66fe-4c31-9763-f1bc2aa459f1/kube-auth-proxy/0.log" Jul 24 14:30:43.165237 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:43.165208 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hhkjn_ebd03052-7832-4502-afd8-41270a2f3a39/payload-pre-processing/0.log" Jul 24 14:30:43.190998 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:43.190963 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:30:44.741099 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:44.741069 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f8885cbd9-fmfxg_809238b9-66fe-4c31-9763-f1bc2aa459f1/kube-auth-proxy/0.log" Jul 24 14:30:44.837919 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:44.837881 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hhkjn_ebd03052-7832-4502-afd8-41270a2f3a39/payload-pre-processing/0.log" Jul 24 14:30:44.875207 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:44.875173 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-spxjx_58b9b0d5-27e3-4158-aa63-c9f89789c786/payload-processing/0.log" Jul 24 14:30:49.209218 ip-10-0-143-193 kubenswrapper[2581]: I0724 14:30:49.209191 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9xkdl_a0b32722-d64a-43ff-b6c7-b19cb05cb829/migrator/0.log"