Sep 14 01:30:54.655347 ip-10-0-130-123 systemd[1]: Starting Kubernetes Kubelet... Sep 14 01:30:55.022751 ip-10-0-130-123 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:55.022751 ip-10-0-130-123 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 01:30:55.022751 ip-10-0-130-123 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:55.022751 ip-10-0-130-123 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 01:30:55.022751 ip-10-0-130-123 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:55.023650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.023564 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 01:30:55.027332 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027317 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.027332 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027332 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027336 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027342 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027345 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027348 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027351 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027354 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027357 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027363 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027366 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027369 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027371 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027374 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027377 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027379 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027382 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027384 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027387 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027390 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027393 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.027395 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027396 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027398 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027401 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027404 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027407 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027410 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027414 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027418 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027422 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027425 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027428 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027430 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027433 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027435 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027438 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027441 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027443 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027446 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027448 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.027869 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027451 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027454 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027456 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027458 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027461 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027463 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027466 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027468 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027471 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027473 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027476 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027478 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027481 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027483 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027486 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027489 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027492 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027495 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027498 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027500 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.028363 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027503 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027506 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027508 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027511 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027513 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027516 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027519 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027521 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027524 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027528 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027532 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027535 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027538 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027541 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027546 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027549 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027552 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027554 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027557 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.028851 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027560 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027563 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027565 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027568 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027570 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027573 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027575 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027578 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027581 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027584 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027586 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027589 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027592 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027594 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027597 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027600 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027602 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027605 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027607 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.029326 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.027610 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.030495 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030483 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.030495 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030494 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030498 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030502 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030506 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030509 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030512 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030519 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030523 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030527 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030531 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030535 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030537 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030540 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030543 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030546 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030549 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030551 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030554 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030557 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.030559 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030560 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030563 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030566 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030568 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030571 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030574 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030577 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030579 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030582 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030584 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030587 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030589 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030592 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030594 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030596 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030599 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030601 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030604 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030607 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.031030 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030609 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030612 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030614 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030617 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030619 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030622 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030626 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030628 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030631 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030633 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030636 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030638 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030640 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030643 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030646 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030649 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030651 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030654 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030657 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030659 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.031509 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030662 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030664 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030667 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030669 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030672 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030675 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030677 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030680 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030683 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030685 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030688 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030690 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030693 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030696 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030698 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030701 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030703 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030706 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030708 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030712 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.031996 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030715 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030717 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030720 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030722 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030725 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030727 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030730 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030733 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030736 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030740 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030743 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030746 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030748 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030751 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030753 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030756 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030758 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030761 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030763 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.032490 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.030766 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031137 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031142 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031145 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031147 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031151 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031153 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031156 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031159 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031162 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031164 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031167 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031170 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031173 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031175 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031178 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031181 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031184 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031186 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031189 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.032948 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031192 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031195 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031197 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031200 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031203 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031205 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031208 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031210 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031213 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031215 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031218 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031220 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031223 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031225 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031228 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031230 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031233 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031236 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031239 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031241 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.033461 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031244 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031248 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031252 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031255 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031257 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031260 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031263 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031265 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031268 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031271 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031273 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031277 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031279 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031283 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031287 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031290 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031293 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031295 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.033945 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031298 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031301 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031304 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031306 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031309 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031312 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031315 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031319 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031321 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031324 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031328 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031331 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031333 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031337 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031339 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031342 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031345 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031348 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031351 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031353 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.034397 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031356 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031358 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031361 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031363 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031366 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031369 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031371 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031374 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031377 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031379 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031382 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031384 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031387 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031389 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031392 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031394 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031397 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031399 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031401 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031404 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.034885 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031406 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.031409 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031482 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031524 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031530 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031534 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031539 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031544 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031548 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031552 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031556 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031559 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031562 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031565 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031568 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031571 2579 flags.go:64] FLAG: --cgroup-root="" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031574 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031578 2579 flags.go:64] FLAG: --client-ca-file="" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031581 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031584 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031588 2579 flags.go:64] FLAG: --cluster-domain="" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031591 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031594 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031597 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 01:30:55.035403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031601 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031605 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031607 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031610 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031614 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031617 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031620 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031623 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031626 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031629 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031634 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031637 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031640 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031643 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031647 2579 flags.go:64] FLAG: --enable-server="true" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031651 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031655 2579 flags.go:64] FLAG: --event-burst="100" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031658 2579 flags.go:64] FLAG: --event-qps="50" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031661 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031664 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031668 2579 flags.go:64] FLAG: --eviction-hard="" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031671 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031674 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031677 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031680 2579 flags.go:64] FLAG: --eviction-soft="" Sep 14 01:30:55.035964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031683 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031686 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031689 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031692 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031695 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031698 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031701 2579 flags.go:64] FLAG: --feature-gates="" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031707 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031710 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031713 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031716 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031719 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031722 2579 flags.go:64] FLAG: --help="false" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031725 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031727 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031730 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031734 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031738 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031741 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031744 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031747 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031750 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031755 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 01:30:55.036572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031758 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031762 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031765 2579 flags.go:64] FLAG: --kube-reserved="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031768 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031771 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031774 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031777 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031779 2579 flags.go:64] FLAG: --lock-file="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031782 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031785 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031788 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031794 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031797 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031800 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031802 2579 flags.go:64] FLAG: --logging-format="text" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031805 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031808 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031811 2579 flags.go:64] FLAG: --manifest-url="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031814 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031819 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031822 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031826 2579 flags.go:64] FLAG: --max-pods="110" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031829 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031832 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031834 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 01:30:55.037200 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031837 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031841 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031844 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031847 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031855 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031858 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031862 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031865 2579 flags.go:64] FLAG: --pod-cidr="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031868 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031874 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031877 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031880 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031883 2579 flags.go:64] FLAG: --port="10250" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031886 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031889 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a32baf3848e45146" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031893 2579 flags.go:64] FLAG: --qos-reserved="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031896 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031899 2579 flags.go:64] FLAG: --register-node="true" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031901 2579 flags.go:64] FLAG: --register-with-taints="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031905 2579 flags.go:64] FLAG: --registry-burst="10" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031908 2579 flags.go:64] FLAG: --registry-qps="5" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031911 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031914 2579 flags.go:64] FLAG: --reserved-memory="" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031918 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031921 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 01:30:55.037825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031924 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031927 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031930 2579 flags.go:64] FLAG: --runonce="false" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031932 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031936 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031939 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031941 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031944 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031949 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031952 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031955 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031958 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031961 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031963 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031967 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031971 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031974 2579 flags.go:64] FLAG: --system-cgroups="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031977 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031982 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031985 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031988 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031992 2579 flags.go:64] FLAG: --tls-min-version="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031995 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.031998 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032001 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 01:30:55.038473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032004 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032007 2579 flags.go:64] FLAG: --v="2" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032011 2579 flags.go:64] FLAG: --version="false" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032014 2579 flags.go:64] FLAG: --vmodule="" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032018 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032021 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032154 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032159 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032162 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032165 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032168 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032170 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032173 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032177 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032181 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032184 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032188 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032190 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032193 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032196 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.039117 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032199 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032205 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032208 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032211 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032215 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032218 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032221 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032224 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032226 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032229 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032232 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032235 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032238 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032240 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032243 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032247 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032251 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032254 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032258 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032261 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.039619 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032264 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032267 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032269 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032272 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032274 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032277 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032280 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032282 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032285 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032287 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032290 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032292 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032295 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032299 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032302 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032304 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032307 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032311 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032313 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032316 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.040128 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032318 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032321 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032324 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032327 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032329 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032332 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032334 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032337 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032340 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032342 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032344 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032347 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032350 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032352 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032355 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032357 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032360 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032362 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032365 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032367 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.040604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032370 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032372 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032375 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032377 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032380 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032385 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032388 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032390 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032393 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032396 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032399 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032401 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032404 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032406 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032409 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032411 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032414 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032416 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032419 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032421 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.041155 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032424 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032426 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032429 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032431 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.032434 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.032440 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.039465 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.039482 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.040198 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.040206 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040282 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040287 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040291 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040295 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040298 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.041651 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040300 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040303 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040306 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040309 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040311 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040314 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040317 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040319 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040323 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040325 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040328 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040330 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040333 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040335 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040338 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040341 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040343 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040346 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040348 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.042125 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040351 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040353 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040356 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040358 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040362 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040372 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040375 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040377 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040380 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040382 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040385 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040389 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040392 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040395 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040398 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040401 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040404 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040407 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040410 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.042722 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040412 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040415 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040418 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040420 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040423 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040425 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040428 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040430 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040434 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040438 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040440 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040443 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040445 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040448 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040450 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040453 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040455 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040458 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040462 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040470 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.043532 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040473 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040475 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040478 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040480 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040483 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040485 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040488 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040491 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040493 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040496 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040499 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040501 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040504 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040506 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040509 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040511 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040514 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040516 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040519 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040521 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.044051 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040524 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040526 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040529 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040531 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040534 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040536 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040539 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040541 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040543 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040546 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040549 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040551 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040559 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040562 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040565 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.044558 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040568 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.040574 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040721 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040726 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040729 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040732 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040735 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040738 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040740 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040743 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040745 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040748 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040751 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040753 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040756 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:55.044918 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040758 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040761 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040765 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040768 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040771 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040773 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040776 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040779 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040781 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040784 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040786 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040789 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040792 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040795 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040797 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040805 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040808 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040810 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040813 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040816 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:55.045305 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040818 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040821 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040823 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040826 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040828 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040831 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040833 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040836 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040838 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040841 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040843 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040846 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040848 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040851 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040853 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040857 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040860 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040863 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040866 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040869 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:55.045782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040872 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040875 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040877 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040880 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040883 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040886 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040889 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040891 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040899 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040902 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040904 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040907 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040909 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040912 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040916 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040919 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040922 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040925 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040928 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:55.046273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040931 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040933 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040936 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040938 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040941 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040943 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040946 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040949 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040951 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040954 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040956 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040959 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040962 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040964 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040967 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040969 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040972 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040974 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040983 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:55.046751 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040985 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040988 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040990 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.040999 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.041002 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.041005 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.041007 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:55.041010 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.041015 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.041149 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.043957 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.044921 2579 server.go:1019] "Starting client certificate rotation" Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.045016 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 01:30:55.047260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.045110 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 01:30:55.047869 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.047856 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 01:30:55.049094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.049077 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 01:30:55.058221 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.058203 2579 log.go:25] "Validated CRI v1 runtime API" Sep 14 01:30:55.062120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.062108 2579 log.go:25] "Validated CRI v1 image API" Sep 14 01:30:55.063006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.062988 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 01:30:55.066978 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.066961 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80fba113-6e92-4e53-ae9f-eaf9795479c1:/dev/nvme0n1p4 abd098ae-e50e-4cd0-bd8d-175641f6eb24:/dev/nvme0n1p3] Sep 14 01:30:55.067036 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.066978 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 01:30:55.072516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.072501 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 01:30:55.072733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.072623 2579 manager.go:216] Machine: {Timestamp:2026-09-14 01:30:55.072133474 +0000 UTC m=+0.305498742 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101225 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bed7540fc318d8eda0dc2d959cd58 SystemUUID:ec2bed75-40fc-318d-8eda-0dc2d959cd58 BootID:2d0cc5d3-4c80-4009-ab41-75e8412498e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:d8:76:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:d8:76:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:2f:27:4d:98:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 01:30:55.072733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.072732 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 01:30:55.072818 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.072805 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 01:30:55.076269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.076247 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 01:30:55.076396 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.076272 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 01:30:55.077092 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.077082 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 01:30:55.077123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.077093 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 01:30:55.077123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.077101 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 01:30:55.077820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.077810 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 01:30:55.078604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.078593 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 01:30:55.083141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083130 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 14 01:30:55.083262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083252 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 01:30:55.083290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083282 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 14 01:30:55.083320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083291 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 01:30:55.083320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083308 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 01:30:55.083320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083317 2579 kubelet.go:398] "Adding apiserver pod source" Sep 14 01:30:55.083437 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083325 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 01:30:55.083785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083753 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 01:30:55.083854 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.083800 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 01:30:55.094731 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.094696 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 01:30:55.094731 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.094696 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 01:30:55.095771 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.095743 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 01:30:55.097638 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.097623 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 01:30:55.097699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.097671 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 01:30:55.099290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099279 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 01:30:55.099330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099295 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 01:30:55.099330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099302 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 01:30:55.099330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099314 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 01:30:55.099330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099326 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 01:30:55.099330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099333 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099347 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099352 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099359 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099364 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099372 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099377 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099399 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 01:30:55.099490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.099405 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 01:30:55.101103 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.101085 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l58zn" Sep 14 01:30:55.102862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.102845 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 01:30:55.103364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.103352 2579 server.go:1268] "Started kubelet" Sep 14 01:30:55.103465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.103416 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 01:30:55.103523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.103440 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 01:30:55.103572 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.103526 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 01:30:55.104376 ip-10-0-130-123 systemd[1]: Started Kubernetes Kubelet. Sep 14 01:30:55.105164 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.105147 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 14 01:30:55.105420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.105398 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 01:30:55.107806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.107788 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l58zn" Sep 14 01:30:55.109507 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.108678 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-123.ec2.internal.18d50bd95a78f52a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-123.ec2.internal,UID:ip-10-0-130-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-123.ec2.internal,},FirstTimestamp:2026-09-14 01:30:55.103325482 +0000 UTC m=+0.336690730,LastTimestamp:2026-09-14 01:30:55.103325482 +0000 UTC m=+0.336690730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-123.ec2.internal,}" Sep 14 01:30:55.109789 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.109755 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 01:30:55.110809 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.110792 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 01:30:55.110888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.110872 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 01:30:55.110969 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.110952 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.111649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.111629 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 01:30:55.111649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.111651 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 01:30:55.111785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.111670 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 01:30:55.111785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.111732 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 14 01:30:55.111785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.111740 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 14 01:30:55.112377 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112362 2579 factory.go:55] Registering systemd factory Sep 14 01:30:55.112484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112466 2579 factory.go:223] Registration of the systemd container factory successfully Sep 14 01:30:55.112691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112675 2579 factory.go:153] Registering CRI-O factory Sep 14 01:30:55.112691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112691 2579 factory.go:223] Registration of the crio container factory successfully Sep 14 01:30:55.112814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112729 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 01:30:55.112814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112741 2579 factory.go:103] Registering Raw factory Sep 14 01:30:55.112814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.112751 2579 manager.go:1196] Started watching for new ooms in manager Sep 14 01:30:55.113220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.113202 2579 manager.go:349] Starting recovery of all containers Sep 14 01:30:55.116761 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.116734 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:55.119931 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.119907 2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-123.ec2.internal\" not found" node="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.124798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.124781 2579 manager.go:354] Recovery completed Sep 14 01:30:55.129582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.129474 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.131304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131291 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.131362 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131319 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.131362 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131333 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.131808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131799 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 01:30:55.131808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131808 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 01:30:55.131887 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131824 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 14 01:30:55.131887 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.131810 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.132517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.132502 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.132578 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.132528 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.132578 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.132537 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.133947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.133930 2579 policy_none.go:49] "None policy: Start" Sep 14 01:30:55.133947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.133946 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 01:30:55.134073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.133957 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 01:30:55.135977 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.135964 2579 policy_none.go:47] "Start" Sep 14 01:30:55.167800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.167781 2579 manager.go:337] "Starting Device Plugin manager" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.167838 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.167852 2579 server.go:87] "Starting device plugin registration server" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.168156 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.168168 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.168258 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.168334 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.168343 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.168859 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 01:30:55.190623 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.168898 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.201720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.201698 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 01:30:55.202864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.202850 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 01:30:55.202930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.202900 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 01:30:55.202930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.202923 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 01:30:55.203017 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.203007 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 01:30:55.204752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.204729 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:55.268780 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.268760 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.270198 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.270181 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.270272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.270217 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.270272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.270233 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.270272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.270261 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.277518 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.277462 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.277518 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.277483 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-123.ec2.internal\": node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.291527 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.291505 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.303961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.303934 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal"] Sep 14 01:30:55.304031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.304003 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.305929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.305912 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.306002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.305944 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.306002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.305959 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.308408 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.308396 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.308523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.308510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.308612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.308551 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.309445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309426 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.309445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309451 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.309577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309466 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.309577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309509 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.309577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309537 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.309577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.309547 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.311792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.311775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.311871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.311805 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:55.312862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.312843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.312960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.312871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.312960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.312893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ac709050433b22e72b53986d3d835e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-123.ec2.internal\" (UID: \"57ac709050433b22e72b53986d3d835e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.313206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.313194 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:55.313267 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.313213 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:55.313267 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.313222 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:55.334680 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.334656 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-123.ec2.internal\" not found" node="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.339026 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.339010 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-123.ec2.internal\" not found" node="ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.392140 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.392116 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.413749 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.413749 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.413888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ac709050433b22e72b53986d3d835e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-123.ec2.internal\" (UID: \"57ac709050433b22e72b53986d3d835e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.413888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57ac709050433b22e72b53986d3d835e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-123.ec2.internal\" (UID: \"57ac709050433b22e72b53986d3d835e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.413888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413811 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.413888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.413840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0fbcb4c87e5de1af39013b60b0a293c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal\" (UID: \"d0fbcb4c87e5de1af39013b60b0a293c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.492934 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.492906 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.593685 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.593621 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.637112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.637073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.642106 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:55.642092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:55.694663 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.694629 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.795127 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.795099 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.895629 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.895558 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:55.996088 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:55.996041 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:56.044501 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.044474 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 01:30:56.044958 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.044632 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 01:30:56.044958 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.044653 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 01:30:56.097046 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:56.097015 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:56.111843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.111802 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 01:25:55 +0000 UTC" deadline="2028-03-26 10:38:43.571898239 +0000 UTC" Sep 14 01:30:56.111843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.111841 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13425h7m47.460060867s" Sep 14 01:30:56.181008 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:56.180979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fbcb4c87e5de1af39013b60b0a293c.slice/crio-24fd8187ed156a636057b109c205ee787c72a5a69b7cd8c4599461a883a0d746 WatchSource:0}: Error finding container 24fd8187ed156a636057b109c205ee787c72a5a69b7cd8c4599461a883a0d746: Status 404 returned error can't find the container with id 24fd8187ed156a636057b109c205ee787c72a5a69b7cd8c4599461a883a0d746 Sep 14 01:30:56.181467 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:56.181440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ac709050433b22e72b53986d3d835e.slice/crio-45f5e1879708a67bd8958e31692f9bead0688f27c4a9fdb4db41c275280103f2 WatchSource:0}: Error finding container 45f5e1879708a67bd8958e31692f9bead0688f27c4a9fdb4db41c275280103f2: Status 404 returned error can't find the container with id 45f5e1879708a67bd8958e31692f9bead0688f27c4a9fdb4db41c275280103f2 Sep 14 01:30:56.188845 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.188830 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:30:56.197438 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:56.197417 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:56.206913 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.206870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" event={"ID":"57ac709050433b22e72b53986d3d835e","Type":"ContainerStarted","Data":"45f5e1879708a67bd8958e31692f9bead0688f27c4a9fdb4db41c275280103f2"} Sep 14 01:30:56.208405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.208383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" event={"ID":"d0fbcb4c87e5de1af39013b60b0a293c","Type":"ContainerStarted","Data":"24fd8187ed156a636057b109c205ee787c72a5a69b7cd8c4599461a883a0d746"} Sep 14 01:30:56.301443 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:56.301404 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-123.ec2.internal\" not found" Sep 14 01:30:56.336290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.336268 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:56.411828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.411797 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" Sep 14 01:30:56.422206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.422138 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 01:30:56.424045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.424031 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" Sep 14 01:30:56.433198 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.433177 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 01:30:56.619943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.619917 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:56.893212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:56.893124 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:57.084563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.084529 2579 apiserver.go:52] "Watching apiserver" Sep 14 01:30:57.091960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.091931 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 01:30:57.093032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.092998 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xhtx9","openshift-ovn-kubernetes/ovnkube-node-wzrrl","kube-system/konnectivity-agent-hr82d","kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal","openshift-dns/node-resolver-szkpn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-5qs9v","openshift-network-diagnostics/network-check-target-lhnbq","openshift-network-operator/iptables-alerter-fjnct","kube-system/global-pull-secret-syncer-q8gjc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m","openshift-cluster-node-tuning-operator/tuned-66mv7","openshift-image-registry/node-ca-jvncq","openshift-multus/multus-pzh7k"] Sep 14 01:30:57.095614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.095591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.095712 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.095681 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:30:57.097804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.097782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.100037 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.100012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.100597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.100464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.100597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.100487 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 01:30:57.100597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.100496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-77946\"" Sep 14 01:30:57.100597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.100466 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.102484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.102456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 01:30:57.102611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.102494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 01:30:57.102611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.102512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t7znq\"" Sep 14 01:30:57.102757 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.102615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.105011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.104847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 01:30:57.105011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.104948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.105180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.105163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7chpb\"" Sep 14 01:30:57.105737 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.105258 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.105737 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.105557 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 01:30:57.105858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.105845 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.107418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.107188 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6sf82\"" Sep 14 01:30:57.107418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.107215 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.107418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.107227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:57.107418 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.107286 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:30:57.107418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.107412 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.109852 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.109612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.111657 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.111639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.112231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.111933 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 01:30:57.112231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.112055 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 01:30:57.112231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.112064 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.112231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.112153 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckrgq\"" Sep 14 01:30:57.112618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.112576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.114846 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.114826 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 01:30:57.114938 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.114837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.115064 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.115045 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c6w2w\"" Sep 14 01:30:57.115523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.115502 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 01:30:57.117075 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.117052 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.117456 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.117399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.117544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.117462 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 01:30:57.117544 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.117463 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:30:57.117660 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.117585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 01:30:57.117964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.117931 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 01:30:57.118097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.118073 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.118178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.118116 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwfkq\"" Sep 14 01:30:57.118178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.118156 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 01:30:57.120264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.120245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.121444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aac06f78-7b86-47c6-a226-4a77d6e7ba68-konnectivity-ca\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.121444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cnibin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-socket-dir-parent\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-daemon-config\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce185961-2578-4eb0-865f-04a5ac136036-host-slash\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-bin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-multus-certs\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-dbus\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-netns\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-system-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-etc-kubernetes\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.121829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2tg\" (UniqueName: \"kubernetes.io/projected/1e9dc715-928b-4cef-ba0d-8cb89739cf45-kube-api-access-np2tg\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ml58\" (UniqueName: \"kubernetes.io/projected/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-kube-api-access-5ml58\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-system-cni-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-tmp-dir\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.121978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce185961-2578-4eb0-865f-04a5ac136036-iptables-alerter-script\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cnibin\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eac84e6-0163-4182-a8ab-63e166b09d6f-host\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-k8s-cni-cncf-io\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-hostroot\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-hosts-file\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z95p\" (UniqueName: \"kubernetes.io/projected/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-kube-api-access-9z95p\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cni-binary-copy\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-conf-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-multus\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aac06f78-7b86-47c6-a226-4a77d6e7ba68-agent-certs\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.122758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-kubelet\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-kubelet-config\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtlk\" (UniqueName: \"kubernetes.io/projected/ce185961-2578-4eb0-865f-04a5ac136036-kube-api-access-5wtlk\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eac84e6-0163-4182-a8ab-63e166b09d6f-serviceca\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbrf\" (UniqueName: \"kubernetes.io/projected/5eac84e6-0163-4182-a8ab-63e166b09d6f-kube-api-access-hcbrf\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.123211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.122935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.125099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.123908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 01:30:57.125099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.124547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4xbn9\"" Sep 14 01:30:57.125099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.124789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.125099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.124875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.127143 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.127090 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 01:30:57.127243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.127192 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwzl\"" Sep 14 01:30:57.127490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.127466 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 01:30:57.127579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.127536 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:30:57.144548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.144506 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-psbpl" Sep 14 01:30:57.149888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.149871 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-psbpl" Sep 14 01:30:57.212890 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.212868 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 01:30:57.223448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87b63fc3-968d-40ea-878e-dc6a1801db68-ovn-node-metrics-cert\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqmd\" (UniqueName: \"kubernetes.io/projected/87b63fc3-968d-40ea-878e-dc6a1801db68-kube-api-access-4bqmd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45h5v\" (UniqueName: \"kubernetes.io/projected/d5450a90-678e-4192-ab47-81dcef253c6c-kube-api-access-45h5v\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ml58\" (UniqueName: \"kubernetes.io/projected/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-kube-api-access-5ml58\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-system-cni-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-systemd-units\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-log-socket\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-system-cni-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-conf\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-tmp-dir\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce185961-2578-4eb0-865f-04a5ac136036-iptables-alerter-script\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cnibin\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-socket-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw5f\" (UniqueName: \"kubernetes.io/projected/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-kube-api-access-9vw5f\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-k8s-cni-cncf-io\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-hostroot\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223889 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cnibin\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z95p\" (UniqueName: \"kubernetes.io/projected/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-kube-api-access-9z95p\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-systemd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.223997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.223986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-hostroot\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-device-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-k8s-cni-cncf-io\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-conf-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-netns\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-var-lib-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-tmp-dir\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-env-overrides\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-registration-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-multus\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-kubelet\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-kubelet-config\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtlk\" (UniqueName: \"kubernetes.io/projected/ce185961-2578-4eb0-865f-04a5ac136036-kube-api-access-5wtlk\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-bin\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce185961-2578-4eb0-865f-04a5ac136036-iptables-alerter-script\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.224628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbrf\" (UniqueName: \"kubernetes.io/projected/5eac84e6-0163-4182-a8ab-63e166b09d6f-kube-api-access-hcbrf\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-slash\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-binary-copy\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-run\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-kubelet\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-multus\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cnibin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-daemon-config\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-conf-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-config\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cnibin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-kubelet-config\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9lb\" (UniqueName: \"kubernetes.io/projected/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kube-api-access-tp9lb\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.225481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-lib-modules\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-host\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-multus-certs\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-node-log\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-var-lib-kubelet\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-netns\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-multus-certs\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-system-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-run-netns\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-etc-kubernetes\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2tg\" (UniqueName: \"kubernetes.io/projected/1e9dc715-928b-4cef-ba0d-8cb89739cf45-kube-api-access-np2tg\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-system-cni-dir\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.224986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysconfig\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-etc-kubernetes\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-systemd\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eac84e6-0163-4182-a8ab-63e166b09d6f-host\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.226356 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.225185 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5eac84e6-0163-4182-a8ab-63e166b09d6f-host\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-daemon-config\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.225270 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.725234537 +0000 UTC m=+2.958599788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-sys\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-hosts-file\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-etc-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-sys-fs\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-hosts-file\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-tmp\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cni-binary-copy\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-kubelet\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-ovn\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.227127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-netd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aac06f78-7b86-47c6-a226-4a77d6e7ba68-agent-certs\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eac84e6-0163-4182-a8ab-63e166b09d6f-serviceca\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-script-lib\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-tuned\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aac06f78-7b86-47c6-a226-4a77d6e7ba68-konnectivity-ca\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.225996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-socket-dir-parent\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e9dc715-928b-4cef-ba0d-8cb89739cf45-cni-binary-copy\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce185961-2578-4eb0-865f-04a5ac136036-host-slash\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-modprobe-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-kubernetes\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce185961-2578-4eb0-865f-04a5ac136036-host-slash\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.227791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-bin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-dbus\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-multus-socket-dir-parent\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226298 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226325 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3e106f2-e5ce-4769-b070-073ce8dee7ac-dbus\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5eac84e6-0163-4182-a8ab-63e166b09d6f-serviceca\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e9dc715-928b-4cef-ba0d-8cb89739cf45-host-var-lib-cni-bin\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.228473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.226637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aac06f78-7b86-47c6-a226-4a77d6e7ba68-konnectivity-ca\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.230408 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.230357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aac06f78-7b86-47c6-a226-4a77d6e7ba68-agent-certs\") pod \"konnectivity-agent-hr82d\" (UID: \"aac06f78-7b86-47c6-a226-4a77d6e7ba68\") " pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.232713 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.232692 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:57.232818 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.232721 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:57.232818 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.232734 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.232818 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.232795 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.7327767 +0000 UTC m=+2.966141940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.233134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.233105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z95p\" (UniqueName: \"kubernetes.io/projected/50a0826b-6cf4-4e18-8a26-6d5e08b1d953-kube-api-access-9z95p\") pod \"multus-additional-cni-plugins-5qs9v\" (UID: \"50a0826b-6cf4-4e18-8a26-6d5e08b1d953\") " pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.233253 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.233148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbrf\" (UniqueName: \"kubernetes.io/projected/5eac84e6-0163-4182-a8ab-63e166b09d6f-kube-api-access-hcbrf\") pod \"node-ca-jvncq\" (UID: \"5eac84e6-0163-4182-a8ab-63e166b09d6f\") " pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.233649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.233595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtlk\" (UniqueName: \"kubernetes.io/projected/ce185961-2578-4eb0-865f-04a5ac136036-kube-api-access-5wtlk\") pod \"iptables-alerter-fjnct\" (UID: \"ce185961-2578-4eb0-865f-04a5ac136036\") " pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.234054 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.234032 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ml58\" (UniqueName: \"kubernetes.io/projected/1006b6cf-ffe6-49bb-afdd-c86e5dd32791-kube-api-access-5ml58\") pod \"node-resolver-szkpn\" (UID: \"1006b6cf-ffe6-49bb-afdd-c86e5dd32791\") " pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.234669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.234649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2tg\" (UniqueName: \"kubernetes.io/projected/1e9dc715-928b-4cef-ba0d-8cb89739cf45-kube-api-access-np2tg\") pod \"multus-pzh7k\" (UID: \"1e9dc715-928b-4cef-ba0d-8cb89739cf45\") " pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.326712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-kubelet\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-ovn\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326735 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-netd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-ovn\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-script-lib\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326808 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-kubelet\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-netd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.326919 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.326902 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.326948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.326992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.82697355 +0000 UTC m=+3.060338786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-tuned\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-modprobe-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-kubernetes\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87b63fc3-968d-40ea-878e-dc6a1801db68-ovn-node-metrics-cert\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqmd\" (UniqueName: \"kubernetes.io/projected/87b63fc3-968d-40ea-878e-dc6a1801db68-kube-api-access-4bqmd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-modprobe-d\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-kubernetes\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45h5v\" (UniqueName: \"kubernetes.io/projected/d5450a90-678e-4192-ab47-81dcef253c6c-kube-api-access-45h5v\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.327295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-systemd-units\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-log-socket\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-conf\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-socket-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw5f\" (UniqueName: \"kubernetes.io/projected/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-kube-api-access-9vw5f\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-log-socket\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-systemd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-systemd-units\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-device-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-netns\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-var-lib-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-socket-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-env-overrides\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-registration-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-bin\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-device-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327670 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysctl-conf\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-slash\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-systemd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-slash\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-run\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-registration-dir\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-config\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-cni-bin\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-host-run-netns\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-run\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9lb\" (UniqueName: \"kubernetes.io/projected/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kube-api-access-tp9lb\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-lib-modules\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.328937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-host\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-var-lib-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-node-log\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.327994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-var-lib-kubelet\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysconfig\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-lib-modules\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-systemd\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-sys\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-etc-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-env-overrides\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-sys-fs\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-sysconfig\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-script-lib\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-etc-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-host\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-tmp\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-sys-fs\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.329584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-node-log\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-systemd\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87b63fc3-968d-40ea-878e-dc6a1801db68-run-openvswitch\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-var-lib-kubelet\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87b63fc3-968d-40ea-878e-dc6a1801db68-ovnkube-config\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.328402 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-sys\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.330112 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:57.330448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.330164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87b63fc3-968d-40ea-878e-dc6a1801db68-ovn-node-metrics-cert\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.330812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.330510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-etc-tuned\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.330812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.330576 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-tmp\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.336731 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.336703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45h5v\" (UniqueName: \"kubernetes.io/projected/d5450a90-678e-4192-ab47-81dcef253c6c-kube-api-access-45h5v\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.336853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.336758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9lb\" (UniqueName: \"kubernetes.io/projected/8f628b1b-cd05-4e6a-aefa-648ac5a1fded-kube-api-access-tp9lb\") pod \"aws-ebs-csi-driver-node-4zf9m\" (UID: \"8f628b1b-cd05-4e6a-aefa-648ac5a1fded\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.337514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.337489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqmd\" (UniqueName: \"kubernetes.io/projected/87b63fc3-968d-40ea-878e-dc6a1801db68-kube-api-access-4bqmd\") pod \"ovnkube-node-wzrrl\" (UID: \"87b63fc3-968d-40ea-878e-dc6a1801db68\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.337768 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.337751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw5f\" (UniqueName: \"kubernetes.io/projected/3ded8f92-d9ca-494c-ae55-f5e223a9e81d-kube-api-access-9vw5f\") pod \"tuned-66mv7\" (UID: \"3ded8f92-d9ca-494c-ae55-f5e223a9e81d\") " pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.409821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.409750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jvncq" Sep 14 01:30:57.416712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.416687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:30:57.426612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.426592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pzh7k" Sep 14 01:30:57.433289 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.433265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szkpn" Sep 14 01:30:57.441968 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.441946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fjnct" Sep 14 01:30:57.450550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.450529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" Sep 14 01:30:57.457411 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.457395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:30:57.464255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.464236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" Sep 14 01:30:57.468936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.468914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-66mv7" Sep 14 01:30:57.731623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.731543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:57.731766 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.731682 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.731766 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.731732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:30:58.731718563 +0000 UTC m=+3.965083798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.832439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.832398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:57.832604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:57.832458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:57.832604 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832522 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:57.832604 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832542 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:57.832604 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832553 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.832604 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832594 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:57.832819 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832610 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:58.832595486 +0000 UTC m=+4.065960729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.832819 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:57.832655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:30:58.832638373 +0000 UTC m=+4.066003610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:58.145517 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.145486 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ded8f92_d9ca_494c_ae55_f5e223a9e81d.slice/crio-7b68ef3cf2ede09e6a8bb8806a7376ace67cd9aa5c8236424c41ebabc609e7ac WatchSource:0}: Error finding container 7b68ef3cf2ede09e6a8bb8806a7376ace67cd9aa5c8236424c41ebabc609e7ac: Status 404 returned error can't find the container with id 7b68ef3cf2ede09e6a8bb8806a7376ace67cd9aa5c8236424c41ebabc609e7ac Sep 14 01:30:58.147450 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.147356 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac06f78_7b86_47c6_a226_4a77d6e7ba68.slice/crio-033f2d962a169faac12acc601731fd0147bb32e1c1c3f93d49c7cb536da8bb4d WatchSource:0}: Error finding container 033f2d962a169faac12acc601731fd0147bb32e1c1c3f93d49c7cb536da8bb4d: Status 404 returned error can't find the container with id 033f2d962a169faac12acc601731fd0147bb32e1c1c3f93d49c7cb536da8bb4d Sep 14 01:30:58.149472 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.149355 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce185961_2578_4eb0_865f_04a5ac136036.slice/crio-e4320d6a0a5f14b55a498609a574ba24bf7b8ab4ae90c023c78e900910fc2020 WatchSource:0}: Error finding container e4320d6a0a5f14b55a498609a574ba24bf7b8ab4ae90c023c78e900910fc2020: Status 404 returned error can't find the container with id e4320d6a0a5f14b55a498609a574ba24bf7b8ab4ae90c023c78e900910fc2020 Sep 14 01:30:58.151941 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.151916 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eac84e6_0163_4182_a8ab_63e166b09d6f.slice/crio-58f46ede0964e2be25c896e1842a995dad79ba0881a6e1e0e6fea13b0660bb0b WatchSource:0}: Error finding container 58f46ede0964e2be25c896e1842a995dad79ba0881a6e1e0e6fea13b0660bb0b: Status 404 returned error can't find the container with id 58f46ede0964e2be25c896e1842a995dad79ba0881a6e1e0e6fea13b0660bb0b Sep 14 01:30:58.153168 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.153087 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9dc715_928b_4cef_ba0d_8cb89739cf45.slice/crio-f498254c3a57dd10ef26e158588aa8612035e0dadc925dc600dd58a3404eef8e WatchSource:0}: Error finding container f498254c3a57dd10ef26e158588aa8612035e0dadc925dc600dd58a3404eef8e: Status 404 returned error can't find the container with id f498254c3a57dd10ef26e158588aa8612035e0dadc925dc600dd58a3404eef8e Sep 14 01:30:58.153853 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.153748 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a0826b_6cf4_4e18_8a26_6d5e08b1d953.slice/crio-519c3ec26d482a6a874321acd8d8a4e88e252ce515f850ef8160b5a5b3d2579f WatchSource:0}: Error finding container 519c3ec26d482a6a874321acd8d8a4e88e252ce515f850ef8160b5a5b3d2579f: Status 404 returned error can't find the container with id 519c3ec26d482a6a874321acd8d8a4e88e252ce515f850ef8160b5a5b3d2579f Sep 14 01:30:58.154208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.154181 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 01:25:57 +0000 UTC" deadline="2028-03-24 00:36:09.038347449 +0000 UTC" Sep 14 01:30:58.154208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.154208 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13367h5m10.884144571s" Sep 14 01:30:58.154896 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.154603 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f628b1b_cd05_4e6a_aefa_648ac5a1fded.slice/crio-af165a30613b350e3fcb92c54aa542ca7aefbb99d0ad8e795664b1b4510cb8e1 WatchSource:0}: Error finding container af165a30613b350e3fcb92c54aa542ca7aefbb99d0ad8e795664b1b4510cb8e1: Status 404 returned error can't find the container with id af165a30613b350e3fcb92c54aa542ca7aefbb99d0ad8e795664b1b4510cb8e1 Sep 14 01:30:58.156208 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.155833 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b63fc3_968d_40ea_878e_dc6a1801db68.slice/crio-d5ae767712009f96151bcacfe3354a092bb1315b5377007725a62924d04dd27d WatchSource:0}: Error finding container d5ae767712009f96151bcacfe3354a092bb1315b5377007725a62924d04dd27d: Status 404 returned error can't find the container with id d5ae767712009f96151bcacfe3354a092bb1315b5377007725a62924d04dd27d Sep 14 01:30:58.157942 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:30:58.157653 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1006b6cf_ffe6_49bb_afdd_c86e5dd32791.slice/crio-b54a7df66b026d8367e5409bd6ac470debf88ab6608e69b551fb412157a5a3c2 WatchSource:0}: Error finding container b54a7df66b026d8367e5409bd6ac470debf88ab6608e69b551fb412157a5a3c2: Status 404 returned error can't find the container with id b54a7df66b026d8367e5409bd6ac470debf88ab6608e69b551fb412157a5a3c2 Sep 14 01:30:58.204471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.204309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:58.204589 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.204562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:30:58.214399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.214370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pzh7k" event={"ID":"1e9dc715-928b-4cef-ba0d-8cb89739cf45","Type":"ContainerStarted","Data":"f498254c3a57dd10ef26e158588aa8612035e0dadc925dc600dd58a3404eef8e"} Sep 14 01:30:58.215639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.215615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvncq" event={"ID":"5eac84e6-0163-4182-a8ab-63e166b09d6f","Type":"ContainerStarted","Data":"58f46ede0964e2be25c896e1842a995dad79ba0881a6e1e0e6fea13b0660bb0b"} Sep 14 01:30:58.216817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.216792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjnct" event={"ID":"ce185961-2578-4eb0-865f-04a5ac136036","Type":"ContainerStarted","Data":"e4320d6a0a5f14b55a498609a574ba24bf7b8ab4ae90c023c78e900910fc2020"} Sep 14 01:30:58.218095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.218072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szkpn" event={"ID":"1006b6cf-ffe6-49bb-afdd-c86e5dd32791","Type":"ContainerStarted","Data":"b54a7df66b026d8367e5409bd6ac470debf88ab6608e69b551fb412157a5a3c2"} Sep 14 01:30:58.219472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.219448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerStarted","Data":"519c3ec26d482a6a874321acd8d8a4e88e252ce515f850ef8160b5a5b3d2579f"} Sep 14 01:30:58.220858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.220829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hr82d" event={"ID":"aac06f78-7b86-47c6-a226-4a77d6e7ba68","Type":"ContainerStarted","Data":"033f2d962a169faac12acc601731fd0147bb32e1c1c3f93d49c7cb536da8bb4d"} Sep 14 01:30:58.222192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.222168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66mv7" event={"ID":"3ded8f92-d9ca-494c-ae55-f5e223a9e81d","Type":"ContainerStarted","Data":"7b68ef3cf2ede09e6a8bb8806a7376ace67cd9aa5c8236424c41ebabc609e7ac"} Sep 14 01:30:58.223530 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.223484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"d5ae767712009f96151bcacfe3354a092bb1315b5377007725a62924d04dd27d"} Sep 14 01:30:58.224646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.224620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" event={"ID":"8f628b1b-cd05-4e6a-aefa-648ac5a1fded","Type":"ContainerStarted","Data":"af165a30613b350e3fcb92c54aa542ca7aefbb99d0ad8e795664b1b4510cb8e1"} Sep 14 01:30:58.740342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.739631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:58.740342 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.739840 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:58.740342 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.739908 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:31:00.739889681 +0000 UTC m=+5.973254918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:58.841093 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.841048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:30:58.841284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:58.841118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:58.841369 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841289 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:58.841369 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841357 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:31:00.841336688 +0000 UTC m=+6.074701952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:58.841819 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841801 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:58.841889 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841827 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:58.841889 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841842 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:58.842009 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:58.841889 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:00.841872692 +0000 UTC m=+6.075237935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:59.155199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.155048 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 01:25:57 +0000 UTC" deadline="2028-02-27 03:37:06.135525474 +0000 UTC" Sep 14 01:30:59.155199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.155134 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12746h6m6.980396026s" Sep 14 01:30:59.206512 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.206479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:30:59.206681 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:59.206629 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:30:59.207059 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.207038 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:30:59.207182 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:30:59.207160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:30:59.239840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.239746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" event={"ID":"57ac709050433b22e72b53986d3d835e","Type":"ContainerStarted","Data":"669e45dc23963190202b52411889438cb813718406efed682c7f7c81d92338fd"} Sep 14 01:30:59.253561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.253379 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0fbcb4c87e5de1af39013b60b0a293c" containerID="e811f9fd874d8df19dd91cffe5d13336efd20074c7a995a9e3e59ac00f8a40f7" exitCode=0 Sep 14 01:30:59.253561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.253421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" event={"ID":"d0fbcb4c87e5de1af39013b60b0a293c","Type":"ContainerDied","Data":"e811f9fd874d8df19dd91cffe5d13336efd20074c7a995a9e3e59ac00f8a40f7"} Sep 14 01:30:59.254310 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:30:59.254088 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-123.ec2.internal" podStartSLOduration=3.254069694 podStartE2EDuration="3.254069694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:30:59.254053691 +0000 UTC m=+4.487418968" watchObservedRunningTime="2026-09-14 01:30:59.254069694 +0000 UTC m=+4.487434957" Sep 14 01:31:00.203988 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.203954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:00.204765 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.204109 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:00.282792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.282758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" event={"ID":"d0fbcb4c87e5de1af39013b60b0a293c","Type":"ContainerStarted","Data":"38702565e946feb1d0eff4c7336fc53bbf5de9ca6cdd9fabb185894caeb135ec"} Sep 14 01:31:00.299681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.299208 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-123.ec2.internal" podStartSLOduration=4.299169342 podStartE2EDuration="4.299169342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:00.298306737 +0000 UTC m=+5.531672024" watchObservedRunningTime="2026-09-14 01:31:00.299169342 +0000 UTC m=+5.532534603" Sep 14 01:31:00.756499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.756460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:00.756685 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.756665 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:00.756764 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.756732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:31:04.756712849 +0000 UTC m=+9.990078100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:00.857770 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.857724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:00.857964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:00.857777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:00.858035 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.857975 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:00.858095 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.858042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:31:04.858021823 +0000 UTC m=+10.091387062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:00.858536 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.858514 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:00.858619 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.858545 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:00.858619 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.858559 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:00.858619 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:00.858612 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:04.858596282 +0000 UTC m=+10.091961742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:01.206104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:01.206004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:01.206617 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:01.206162 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:01.206617 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:01.206194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:01.206617 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:01.206276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:02.203382 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:02.203326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:02.203568 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:02.203459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:03.203831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:03.203370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:03.203831 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:03.203495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:03.203831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:03.203829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:03.204417 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:03.203940 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:04.203973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:04.203891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:04.204329 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.204020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:04.793038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:04.792501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:04.793038 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.792652 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:04.793038 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.792713 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:31:12.79269615 +0000 UTC m=+18.026061393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:04.893443 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:04.893401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:04.893621 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:04.893456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:04.893621 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.893593 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:04.893728 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.893650 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:31:12.893630515 +0000 UTC m=+18.126995765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:04.894093 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.894075 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:04.894180 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.894097 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:04.894180 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.894110 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:04.894180 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:04.894155 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:12.894139785 +0000 UTC m=+18.127505022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:05.204697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:05.204609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:05.205120 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:05.204732 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:05.205120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:05.205095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:05.205243 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:05.205190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:06.203891 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:06.203408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:06.203891 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:06.203528 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:07.203430 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:07.203394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:07.203874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:07.203450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:07.203874 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:07.203550 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:07.203874 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:07.203675 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:08.203983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:08.203949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:08.204422 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:08.204101 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:09.203573 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:09.203539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:09.203754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:09.203543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:09.203754 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:09.203697 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:09.203846 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:09.203760 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:10.203349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:10.203318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:10.203754 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:10.203422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:11.203196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:11.203168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:11.203196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:11.203190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:11.203419 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:11.203326 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:11.203730 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:11.203447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:12.204052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:12.204022 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:12.204427 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.204129 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:12.845226 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:12.845194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:12.845377 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.845297 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:12.845377 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.845355 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:31:28.845340855 +0000 UTC m=+34.078706090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:12.945754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:12.945720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:12.945754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:12.945758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945890 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945903 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945926 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945937 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945944 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:31:28.945929856 +0000 UTC m=+34.179295091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:12.945981 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:12.945984 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:28.945967655 +0000 UTC m=+34.179332912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:13.203346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:13.203274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:13.203504 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:13.203422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:13.203504 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:13.203482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:13.203644 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:13.203606 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:14.204186 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:14.203991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:14.204618 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:14.204273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:15.204556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:15.204525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:15.204980 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:15.204623 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:15.204980 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:15.204691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:15.204980 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:15.204775 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:16.203654 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.203589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:16.203748 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:16.203697 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:16.317326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.317102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-66mv7" event={"ID":"3ded8f92-d9ca-494c-ae55-f5e223a9e81d","Type":"ContainerStarted","Data":"881865553a78e5eed15bb5b4860e98507cec66b2676a599ac8a6be0a6daf1ff9"} Sep 14 01:31:16.318961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.318934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"590bfb369d6a4f3b1132c46466417f93aaa1141f2c3b36e1b6c062d744965b40"} Sep 14 01:31:16.319080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.319013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"3bcb1cc2a3c673f2af9e021f6608dca697ab40db394d43a12a194f49f5444087"} Sep 14 01:31:16.320614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.320591 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" event={"ID":"8f628b1b-cd05-4e6a-aefa-648ac5a1fded","Type":"ContainerStarted","Data":"9a7ca803ca27336ae4d3bb7275d81057ecf27e6e4b76ba0fa4fa4bf0238d1bcf"} Sep 14 01:31:16.322109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.322087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pzh7k" event={"ID":"1e9dc715-928b-4cef-ba0d-8cb89739cf45","Type":"ContainerStarted","Data":"0363ae58d0071c34e9076fa3b93181937e2361e19890cb3fdac402eff57277c4"} Sep 14 01:31:16.323581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.323560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jvncq" event={"ID":"5eac84e6-0163-4182-a8ab-63e166b09d6f","Type":"ContainerStarted","Data":"f77e75c6d4ea6ca0c74c1074f2b3f76a162e14f42814c80d867965977d3140f0"} Sep 14 01:31:16.325428 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.325403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szkpn" event={"ID":"1006b6cf-ffe6-49bb-afdd-c86e5dd32791","Type":"ContainerStarted","Data":"0d49dd62520ee0e787bb9621575cb1a5d4e65046a8f0e55252dfaf94f717b10f"} Sep 14 01:31:16.327067 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.327031 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="4950c8f18a026806d64b316b26570a93e8165e5855378aacf7d971c888a4e01a" exitCode=0 Sep 14 01:31:16.327152 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.327099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"4950c8f18a026806d64b316b26570a93e8165e5855378aacf7d971c888a4e01a"} Sep 14 01:31:16.328790 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.328761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hr82d" event={"ID":"aac06f78-7b86-47c6-a226-4a77d6e7ba68","Type":"ContainerStarted","Data":"438f322fb3be5d032ca66e484a879dcf696ffeb0875178bd0901c523221d0e73"} Sep 14 01:31:16.332150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.332111 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-66mv7" podStartSLOduration=3.46741152 podStartE2EDuration="21.332098323s" totalImagesPullingTime="17.864686803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.33183297 +0000 UTC m=+21.565198305" watchObservedRunningTime="2026-09-14 01:31:16.332098323 +0000 UTC m=+21.565463579" Sep 14 01:31:16.349413 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.349358 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jvncq" podStartSLOduration=3.492259926 podStartE2EDuration="21.349340969s" totalImagesPullingTime="17.857081043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.348889289 +0000 UTC m=+21.582254566" watchObservedRunningTime="2026-09-14 01:31:16.349340969 +0000 UTC m=+21.582706227" Sep 14 01:31:16.363431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.363376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hr82d" podStartSLOduration=10.860459709 podStartE2EDuration="21.363364889s" totalImagesPullingTime="10.50290518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.363198675 +0000 UTC m=+21.596563932" watchObservedRunningTime="2026-09-14 01:31:16.363364889 +0000 UTC m=+21.596730146" Sep 14 01:31:16.402592 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.402551 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pzh7k" podStartSLOduration=3.496016143 podStartE2EDuration="21.402535957s" totalImagesPullingTime="17.906519814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.402392665 +0000 UTC m=+21.635757941" watchObservedRunningTime="2026-09-14 01:31:16.402535957 +0000 UTC m=+21.635901214" Sep 14 01:31:16.411800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.411779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:16.417039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.416980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-szkpn" podStartSLOduration=3.566093661 podStartE2EDuration="21.416962547s" totalImagesPullingTime="17.850868886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.416417857 +0000 UTC m=+21.649783139" watchObservedRunningTime="2026-09-14 01:31:16.416962547 +0000 UTC m=+21.650327804" Sep 14 01:31:16.466445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.466425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:16.567550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:16.567530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:17.143662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.143623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:17.203666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.203638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:17.203838 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:17.203767 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:17.203838 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.203802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:17.203950 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:17.203907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:17.243419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.243388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:17.333993 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.333964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"352229c69a19b691012f37d5cd3b8208419a612e03ef0cacafa7c177cd3f46ca"} Sep 14 01:31:17.334483 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.334032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"63fff10113a052cce07fa1350cc677185f392d6784e047db081891ddad38e8ff"} Sep 14 01:31:17.334483 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.334084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"cb850753fb7f667a62d7da7de24b3db16334c09770f479dc5babb39b9ebee806"} Sep 14 01:31:17.334483 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.334116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"a25b6e2bfeada71fe95941b5a89b00bd06f05a1f8084bde4d0aff48b809ae44e"} Sep 14 01:31:17.457943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.457920 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 01:31:17.545246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.545220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:17.600625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:17.600549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:18.184981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.184854 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T01:31:17.457938633Z","UUID":"e7e9a50c-4029-43af-aa2d-f21c1a9be672","Handler":null,"Name":"","Endpoint":""} Sep 14 01:31:18.186729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.186705 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 01:31:18.186909 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.186743 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 01:31:18.203636 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.203599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:18.203791 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:18.203723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:18.232791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.232745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:18.333871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.333818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:18.337125 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.337101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" event={"ID":"8f628b1b-cd05-4e6a-aefa-648ac5a1fded","Type":"ContainerStarted","Data":"9ff460870459a6c10b9d9b136134fac4c2e6b6282a0612a507e9b04fb388089d"} Sep 14 01:31:18.338418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.338397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fjnct" event={"ID":"ce185961-2578-4eb0-865f-04a5ac136036","Type":"ContainerStarted","Data":"47eacfad915461390373aa09027acad3ba73ae51417bd4bee54d88429b5ddd06"} Sep 14 01:31:18.352548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.352506 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fjnct" podStartSLOduration=5.493270163 podStartE2EDuration="23.352493528s" totalImagesPullingTime="17.859223365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:18.351963618 +0000 UTC m=+23.585328875" watchObservedRunningTime="2026-09-14 01:31:18.352493528 +0000 UTC m=+23.585858784" Sep 14 01:31:18.433817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.433779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:18.750730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.750704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:18.850555 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:18.850524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:19.203986 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.203957 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:19.204259 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:19.204152 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:19.204259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.204209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:19.204381 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:19.204329 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:19.344708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.344667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"04ac79760c752779ac149e554791e536e715f6fcad01197d8fef3d9c3b2e8355"} Sep 14 01:31:19.347117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.347083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" event={"ID":"8f628b1b-cd05-4e6a-aefa-648ac5a1fded","Type":"ContainerStarted","Data":"9b32c702ba1e6aef024d725d6a2d3b68dfab4837b20618b5b5e4aa950942c2a2"} Sep 14 01:31:19.363551 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.363502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zf9m" podStartSLOduration=4.421620134 podStartE2EDuration="24.363484863s" totalImagesPullingTime="19.941864729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:19.362200713 +0000 UTC m=+24.595566007" watchObservedRunningTime="2026-09-14 01:31:19.363484863 +0000 UTC m=+24.596850149" Sep 14 01:31:19.544330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.544231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:19.775188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.775152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:19.828330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.828250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:19.946567 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:19.946534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:20.049199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:20.049167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:20.203503 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:20.203434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:20.203639 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:20.203548 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:20.443677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:20.443647 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:31:20.444380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:20.444361 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:31:21.162072 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.162040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:21.203437 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.203415 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:21.203534 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:21.203521 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:21.203595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.203575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:21.203707 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:21.203688 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:21.353979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.353717 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="0793cab10c0c578e529b97e33daf6b52db411fb57845956e3242bd1508f0d09b" exitCode=0 Sep 14 01:31:21.354135 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.353793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"0793cab10c0c578e529b97e33daf6b52db411fb57845956e3242bd1508f0d09b"} Sep 14 01:31:21.358115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.358089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" event={"ID":"87b63fc3-968d-40ea-878e-dc6a1801db68","Type":"ContainerStarted","Data":"c7a429379ddfc309a2b584ac337b5c35ce530d85368b757d3bef0f1c6fcffb8f"} Sep 14 01:31:21.398846 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:21.397851 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" podStartSLOduration=8.503420811 podStartE2EDuration="26.397834656s" totalImagesPullingTime="17.894413845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:21.397086723 +0000 UTC m=+26.630451981" watchObservedRunningTime="2026-09-14 01:31:21.397834656 +0000 UTC m=+26.631199915" Sep 14 01:31:22.203788 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.203760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:22.204210 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:22.203852 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:22.361589 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.361564 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:22.361733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.361631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:22.361733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.361660 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:22.380405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.380363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:22.380573 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.380554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:22.920873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.920843 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q8gjc"] Sep 14 01:31:22.921045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.920948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:22.921045 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:22.921032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:22.924311 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.924281 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lhnbq"] Sep 14 01:31:22.924431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.924389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:22.924532 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:22.924486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:22.924925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.924903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xhtx9"] Sep 14 01:31:22.925027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:22.925015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:22.925156 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:22.925135 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:23.365425 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:23.365392 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="41757e117b3a14e3bb5df30905e5706459a0a5214ed01517c348ed3bb2329593" exitCode=0 Sep 14 01:31:23.366166 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:23.365483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"41757e117b3a14e3bb5df30905e5706459a0a5214ed01517c348ed3bb2329593"} Sep 14 01:31:24.204017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:24.203949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:24.204170 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:24.204048 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:25.203969 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:25.203752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:25.203969 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:25.203961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:25.204406 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:25.203817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:25.204406 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:25.204094 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:25.371960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:25.371931 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="036bac209899b52b9ec4a3eedc95c9283f3a8bca26dea1877643a29b3074ec83" exitCode=0 Sep 14 01:31:25.372124 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:25.371990 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"036bac209899b52b9ec4a3eedc95c9283f3a8bca26dea1877643a29b3074ec83"} Sep 14 01:31:26.203797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:26.203767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:26.203995 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:26.203973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:26.836796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:26.836758 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:31:26.836985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:26.836905 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 14 01:31:26.837389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:26.837371 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hr82d" Sep 14 01:31:26.890391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:26.890356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:27.203571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:27.203495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:27.203719 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:27.203658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:27.203785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:27.203719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:27.203853 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:27.203832 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:28.203174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.203096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:28.203629 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.203217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:31:28.756120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.756045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:28.804690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.804656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:28.870039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.869999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:28.870231 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.870140 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:28.870231 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.870216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret podName:c3e106f2-e5ce-4769-b070-073ce8dee7ac nodeName:}" failed. No retries permitted until 2026-09-14 01:32:00.870197491 +0000 UTC m=+66.103562730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret") pod "global-pull-secret-syncer-q8gjc" (UID: "c3e106f2-e5ce-4769-b070-073ce8dee7ac") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:28.970463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.970426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:28.970463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:28.970469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:28.970700 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970592 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:28.970700 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970615 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:28.970700 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970640 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:28.970700 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970652 2579 projected.go:196] Error preparing data for projected volume kube-api-access-87s8c for pod openshift-network-diagnostics/network-check-target-lhnbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:28.970700 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970661 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:32:00.970640715 +0000 UTC m=+66.204005966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:28.970938 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:28.970700 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c podName:c0241056-be71-439b-9ea3-9c1419737a26 nodeName:}" failed. No retries permitted until 2026-09-14 01:32:00.970682824 +0000 UTC m=+66.204048079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87s8c" (UniqueName: "kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c") pod "network-check-target-lhnbq" (UID: "c0241056-be71-439b-9ea3-9c1419737a26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:29.203881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:29.203794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:29.203881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:29.203830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:29.204325 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:29.203943 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhnbq" podUID="c0241056-be71-439b-9ea3-9c1419737a26" Sep 14 01:31:29.204325 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:29.203962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q8gjc" podUID="c3e106f2-e5ce-4769-b070-073ce8dee7ac" Sep 14 01:31:29.386602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:29.386569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerStarted","Data":"f60086355f451b6a0284cd0bb208da392edf52a199d7a2ae1339a5522e79642a"} Sep 14 01:31:30.091995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.091969 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-123.ec2.internal" event="NodeReady" Sep 14 01:31:30.092178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.092093 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 01:31:30.136520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.136485 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4vzjs"] Sep 14 01:31:30.139284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.139257 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n646v"] Sep 14 01:31:30.139410 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.139381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.141971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.141946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 01:31:30.142175 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.142157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jfmw6\"" Sep 14 01:31:30.142175 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.142165 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 01:31:30.142464 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.142447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.145033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.145015 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 01:31:30.145142 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.145038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 01:31:30.145203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.145174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tbbp2\"" Sep 14 01:31:30.145260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.145226 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 01:31:30.147415 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.147396 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vzjs"] Sep 14 01:31:30.150577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.150559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n646v"] Sep 14 01:31:30.203301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.203273 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:31:30.205612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.205591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 01:31:30.205971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.205641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gv4zq\"" Sep 14 01:31:30.244820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.244796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:30.281908 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.281885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.281976 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.281937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/685a7a90-19bb-49dc-b5df-54ae3937a55a-kube-api-access-52cw8\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.282014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.281979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685a7a90-19bb-49dc-b5df-54ae3937a55a-config-volume\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.282014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.282007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685a7a90-19bb-49dc-b5df-54ae3937a55a-tmp-dir\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.282105 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.282039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6153521d-a4f5-4e75-9ce8-c477938c92d5-cert\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.282137 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.282109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52vkm\" (UniqueName: \"kubernetes.io/projected/6153521d-a4f5-4e75-9ce8-c477938c92d5-kube-api-access-52vkm\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.343802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.343733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:30.383120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.383263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/685a7a90-19bb-49dc-b5df-54ae3937a55a-kube-api-access-52cw8\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.383263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685a7a90-19bb-49dc-b5df-54ae3937a55a-config-volume\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.383263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685a7a90-19bb-49dc-b5df-54ae3937a55a-tmp-dir\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.383263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6153521d-a4f5-4e75-9ce8-c477938c92d5-cert\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.383263 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:30.383232 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:30.383475 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:30.383299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:31:30.883276679 +0000 UTC m=+36.116641928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:30.383475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52vkm\" (UniqueName: \"kubernetes.io/projected/6153521d-a4f5-4e75-9ce8-c477938c92d5-kube-api-access-52vkm\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.383602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685a7a90-19bb-49dc-b5df-54ae3937a55a-tmp-dir\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.383712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.383695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/685a7a90-19bb-49dc-b5df-54ae3937a55a-config-volume\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.387327 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.387306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6153521d-a4f5-4e75-9ce8-c477938c92d5-cert\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.391647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.391620 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="f60086355f451b6a0284cd0bb208da392edf52a199d7a2ae1339a5522e79642a" exitCode=0 Sep 14 01:31:30.391745 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.391663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"f60086355f451b6a0284cd0bb208da392edf52a199d7a2ae1339a5522e79642a"} Sep 14 01:31:30.393388 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.393363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cw8\" (UniqueName: \"kubernetes.io/projected/685a7a90-19bb-49dc-b5df-54ae3937a55a-kube-api-access-52cw8\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.393600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.393582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52vkm\" (UniqueName: \"kubernetes.io/projected/6153521d-a4f5-4e75-9ce8-c477938c92d5-kube-api-access-52vkm\") pod \"ingress-canary-n646v\" (UID: \"6153521d-a4f5-4e75-9ce8-c477938c92d5\") " pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.454954 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.454930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n646v" Sep 14 01:31:30.732196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.731962 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n646v"] Sep 14 01:31:30.735225 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:31:30.735201 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6153521d_a4f5_4e75_9ce8_c477938c92d5.slice/crio-2ccba11f690884f5090c8daa418c80823c9cfe7e846ad92863a6d05df6d8166c WatchSource:0}: Error finding container 2ccba11f690884f5090c8daa418c80823c9cfe7e846ad92863a6d05df6d8166c: Status 404 returned error can't find the container with id 2ccba11f690884f5090c8daa418c80823c9cfe7e846ad92863a6d05df6d8166c Sep 14 01:31:30.887046 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:30.887013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:30.887196 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:30.887154 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:30.887234 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:30.887216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:31:31.887199991 +0000 UTC m=+37.120565226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:31.204126 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.204027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:31:31.204126 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.204088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:31:31.206686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.206663 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 01:31:31.207291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.206730 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 01:31:31.207291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.206899 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 01:31:31.207291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.207000 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tck5c\"" Sep 14 01:31:31.398097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.398047 2579 generic.go:356] "Generic (PLEG): container finished" podID="50a0826b-6cf4-4e18-8a26-6d5e08b1d953" containerID="1d166eb149f6d5bae49042decc358181680c7064bc74dde40bf6d96cb3fed826" exitCode=0 Sep 14 01:31:31.398219 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.398096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerDied","Data":"1d166eb149f6d5bae49042decc358181680c7064bc74dde40bf6d96cb3fed826"} Sep 14 01:31:31.399427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.399406 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n646v" event={"ID":"6153521d-a4f5-4e75-9ce8-c477938c92d5","Type":"ContainerStarted","Data":"2ccba11f690884f5090c8daa418c80823c9cfe7e846ad92863a6d05df6d8166c"} Sep 14 01:31:31.893170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:31.893094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:31.893321 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:31.893259 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:31.893372 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:31.893331 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:31:33.89330912 +0000 UTC m=+39.126674359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:32.312893 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.312800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:32.406830 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.406792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" event={"ID":"50a0826b-6cf4-4e18-8a26-6d5e08b1d953","Type":"ContainerStarted","Data":"75da6dfc40e68e9ed411c127b275f2bf0eb0cee178526a8822a829229d2c1955"} Sep 14 01:31:32.412187 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.412133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:32.427169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.427130 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5qs9v" podStartSLOduration=8.956679326 podStartE2EDuration="37.427114132s" totalImagesPullingTime="28.470434806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:32.426438885 +0000 UTC m=+37.659804156" watchObservedRunningTime="2026-09-14 01:31:32.427114132 +0000 UTC m=+37.660479391" Sep 14 01:31:32.613515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.613419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:32.667720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:32.667673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:33.411279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:33.411085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n646v" event={"ID":"6153521d-a4f5-4e75-9ce8-c477938c92d5","Type":"ContainerStarted","Data":"e29989451558bde9521dc52b9475778fb5a13b483c3d527df384b32c4f88fba8"} Sep 14 01:31:33.426381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:33.426333 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n646v" podStartSLOduration=1.1417388179999999 podStartE2EDuration="3.426319688s" totalImagesPullingTime="2.28458087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:33.426131106 +0000 UTC m=+38.659496380" watchObservedRunningTime="2026-09-14 01:31:33.426319688 +0000 UTC m=+38.659684966" Sep 14 01:31:33.660380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:33.660350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:33.761338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:33.761311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:33.908308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:33.908278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:33.908464 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:33.908373 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:33.908464 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:33.908423 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:31:37.908409805 +0000 UTC m=+43.141775040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:34.356440 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:34.356405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:34.403039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:34.403000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:37.935517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:37.935486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:37.935875 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:37.935598 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:37.935875 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:37.935650 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:31:45.935636735 +0000 UTC m=+51.169001970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:42.650246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:42.650211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:42.698342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:42.698313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:45.990076 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:45.990014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:31:45.990569 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:45.990191 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:45.990569 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:31:45.990308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:32:01.99027906 +0000 UTC m=+67.223644306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:31:54.377553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:54.377524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzrrl" Sep 14 01:31:54.533142 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:54.533106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:54.591399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:54.591367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:55.250782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:55.250752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:55.351264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:31:55.351232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:32:00.886031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.885990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:32:00.888412 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.888393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 01:32:00.900475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.900456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3e106f2-e5ce-4769-b070-073ce8dee7ac-original-pull-secret\") pod \"global-pull-secret-syncer-q8gjc\" (UID: \"c3e106f2-e5ce-4769-b070-073ce8dee7ac\") " pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:32:00.917662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.917640 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q8gjc" Sep 14 01:32:00.986510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.986478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:32:00.986651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.986519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:32:00.988785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.988743 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 01:32:00.988929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.988912 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 01:32:00.997348 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:00.997329 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:32:00.997453 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:00.997412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:33:04.997389943 +0000 UTC m=+130.230755179 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : secret "metrics-daemon-secret" not found Sep 14 01:32:00.999307 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:00.999290 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 01:32:01.009838 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.009817 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87s8c\" (UniqueName: \"kubernetes.io/projected/c0241056-be71-439b-9ea3-9c1419737a26-kube-api-access-87s8c\") pod \"network-check-target-lhnbq\" (UID: \"c0241056-be71-439b-9ea3-9c1419737a26\") " pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:32:01.181090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.180997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q8gjc"] Sep 14 01:32:01.184321 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:32:01.184300 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e106f2_e5ce_4769_b070_073ce8dee7ac.slice/crio-5bd9e43ee488efa7b4164964c8787aa317753dc2eeec9f72088c04b63718b9d0 WatchSource:0}: Error finding container 5bd9e43ee488efa7b4164964c8787aa317753dc2eeec9f72088c04b63718b9d0: Status 404 returned error can't find the container with id 5bd9e43ee488efa7b4164964c8787aa317753dc2eeec9f72088c04b63718b9d0 Sep 14 01:32:01.223843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.223819 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tck5c\"" Sep 14 01:32:01.232703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.232685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:32:01.476844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.476773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q8gjc" event={"ID":"c3e106f2-e5ce-4769-b070-073ce8dee7ac","Type":"ContainerStarted","Data":"5bd9e43ee488efa7b4164964c8787aa317753dc2eeec9f72088c04b63718b9d0"} Sep 14 01:32:01.495561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.495535 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lhnbq"] Sep 14 01:32:01.498303 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:32:01.498279 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0241056_be71_439b_9ea3_9c1419737a26.slice/crio-ca1b022239206f4cfeb6fe6607e670075be0ff5556fb91c10614143b73df3b46 WatchSource:0}: Error finding container ca1b022239206f4cfeb6fe6607e670075be0ff5556fb91c10614143b73df3b46: Status 404 returned error can't find the container with id ca1b022239206f4cfeb6fe6607e670075be0ff5556fb91c10614143b73df3b46 Sep 14 01:32:01.993312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:01.993270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:32:01.993771 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:01.993443 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:32:01.993771 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:01.993519 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:32:33.993498682 +0000 UTC m=+99.226863924 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:32:02.481515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:02.481473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lhnbq" event={"ID":"c0241056-be71-439b-9ea3-9c1419737a26","Type":"ContainerStarted","Data":"ca1b022239206f4cfeb6fe6607e670075be0ff5556fb91c10614143b73df3b46"} Sep 14 01:32:06.493805 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:06.493762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q8gjc" event={"ID":"c3e106f2-e5ce-4769-b070-073ce8dee7ac","Type":"ContainerStarted","Data":"f5e04934a2a1362a807cabc7f432cada6053d9ea450103bd55fddb4b3b6e4744"} Sep 14 01:32:06.495176 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:06.495152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lhnbq" event={"ID":"c0241056-be71-439b-9ea3-9c1419737a26","Type":"ContainerStarted","Data":"d20d0d8820b4a07e18dbf1f1e14416aa7fee6a4b19b95e396e8f80a1da500313"} Sep 14 01:32:06.495284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:06.495203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:32:06.508317 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:06.508275 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q8gjc" podStartSLOduration=66.994802592 podStartE2EDuration="1m11.508263898s" totalImagesPullingTime="4.513461306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:06.507469328 +0000 UTC m=+71.740834596" watchObservedRunningTime="2026-09-14 01:32:06.508263898 +0000 UTC m=+71.741629155" Sep 14 01:32:06.521040 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:06.521003 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lhnbq" podStartSLOduration=67.323788897 podStartE2EDuration="1m11.520991811s" totalImagesPullingTime="4.197202914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:06.520618934 +0000 UTC m=+71.753984190" watchObservedRunningTime="2026-09-14 01:32:06.520991811 +0000 UTC m=+71.754357067" Sep 14 01:32:19.606775 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:19.606657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:32:19.660176 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:19.660146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:32:34.092686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:34.092628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:32:34.093093 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:34.092784 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:32:34.093093 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:32:34.092851 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:33:38.092834795 +0000 UTC m=+163.326200030 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:32:36.114336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.114299 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc"] Sep 14 01:32:36.118249 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.118231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.121089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.121072 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-frvqb\"" Sep 14 01:32:36.121195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.121095 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 01:32:36.121195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.121098 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 01:32:36.122396 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.122380 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 01:32:36.122479 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.122379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 14 01:32:36.124740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.124720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc"] Sep 14 01:32:36.307045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.307007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.307233 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.307174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggv8\" (UniqueName: \"kubernetes.io/projected/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-kube-api-access-mggv8\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.408385 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.408306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.408385 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.408344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggv8\" (UniqueName: \"kubernetes.io/projected/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-kube-api-access-mggv8\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.410894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.410856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.416109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.416089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggv8\" (UniqueName: \"kubernetes.io/projected/3def828b-cbfd-4b81-bf5a-8af68a41c3d7-kube-api-access-mggv8\") pod \"managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc\" (UID: \"3def828b-cbfd-4b81-bf5a-8af68a41c3d7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.437491 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.437468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" Sep 14 01:32:36.702256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:36.702184 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc"] Sep 14 01:32:36.705358 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:32:36.705332 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3def828b_cbfd_4b81_bf5a_8af68a41c3d7.slice/crio-efd7830f9f7a9da61c122d3f7f44c8083a3a76bd21d6e77e40744232bf63ffff WatchSource:0}: Error finding container efd7830f9f7a9da61c122d3f7f44c8083a3a76bd21d6e77e40744232bf63ffff: Status 404 returned error can't find the container with id efd7830f9f7a9da61c122d3f7f44c8083a3a76bd21d6e77e40744232bf63ffff Sep 14 01:32:37.499783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:37.499755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lhnbq" Sep 14 01:32:37.571422 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:37.571352 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" event={"ID":"3def828b-cbfd-4b81-bf5a-8af68a41c3d7","Type":"ContainerStarted","Data":"efd7830f9f7a9da61c122d3f7f44c8083a3a76bd21d6e77e40744232bf63ffff"} Sep 14 01:32:40.583689 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:40.583654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" event={"ID":"3def828b-cbfd-4b81-bf5a-8af68a41c3d7","Type":"ContainerStarted","Data":"6eb1bbc2c848517ab43778d88057e2f02ab547766f74faee018c5464c059f9e0"} Sep 14 01:32:40.597426 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:32:40.597386 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" podStartSLOduration=1.556608787 podStartE2EDuration="4.597373274s" totalImagesPullingTime="3.040764487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:40.597290916 +0000 UTC m=+105.830656199" watchObservedRunningTime="2026-09-14 01:32:40.597373274 +0000 UTC m=+105.830738531" Sep 14 01:33:05.090863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:33:05.090823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:33:05.091273 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:05.090938 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:33:05.091273 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:05.091000 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:35:07.090981075 +0000 UTC m=+252.324346331 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : secret "metrics-daemon-secret" not found Sep 14 01:33:33.150464 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:33.150420 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vzjs" podUID="685a7a90-19bb-49dc-b5df-54ae3937a55a" Sep 14 01:33:33.211355 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:33.211320 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:33:33.710475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:33:33.710447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:33:38.115497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:33:38.115461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:33:38.115885 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:38.115619 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:33:38.115885 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:33:38.115697 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls podName:685a7a90-19bb-49dc-b5df-54ae3937a55a nodeName:}" failed. No retries permitted until 2026-09-14 01:35:40.115678539 +0000 UTC m=+285.349043782 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls") pod "dns-default-4vzjs" (UID: "685a7a90-19bb-49dc-b5df-54ae3937a55a") : secret "dns-default-metrics-tls" not found Sep 14 01:33:47.203874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:33:47.203833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:34:40.867573 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:34:40.867534 2579 generic.go:356] "Generic (PLEG): container finished" podID="3def828b-cbfd-4b81-bf5a-8af68a41c3d7" containerID="6eb1bbc2c848517ab43778d88057e2f02ab547766f74faee018c5464c059f9e0" exitCode=255 Sep 14 01:34:40.867573 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:34:40.867573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" event={"ID":"3def828b-cbfd-4b81-bf5a-8af68a41c3d7","Type":"ContainerDied","Data":"6eb1bbc2c848517ab43778d88057e2f02ab547766f74faee018c5464c059f9e0"} Sep 14 01:34:40.873017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:34:40.872999 2579 scope.go:117] "RemoveContainer" containerID="6eb1bbc2c848517ab43778d88057e2f02ab547766f74faee018c5464c059f9e0" Sep 14 01:34:41.871720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:34:41.871690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76fcd88f6d-k7cqc" event={"ID":"3def828b-cbfd-4b81-bf5a-8af68a41c3d7","Type":"ContainerStarted","Data":"a58d44559c42ee4bd75b3541eb68a210f9c2b17f46be9ff80ef2a551caac27f0"} Sep 14 01:35:04.574638 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.574601 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:04.577510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.577495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.579905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.579886 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 01:35:04.580238 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.580222 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 01:35:04.580432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.580414 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh46f\"" Sep 14 01:35:04.580508 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.580450 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 01:35:04.585273 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.585251 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 01:35:04.592004 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.591982 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:04.662436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.662402 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99"] Sep 14 01:35:04.665224 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.665205 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh"] Sep 14 01:35:04.665361 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.665340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" Sep 14 01:35:04.667863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.667840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.668155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.668134 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-md2qs\"" Sep 14 01:35:04.668328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.668309 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:35:04.668544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.668520 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 14 01:35:04.669183 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.669168 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44"] Sep 14 01:35:04.673329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.673308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2lks5\"" Sep 14 01:35:04.673523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.673504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:35:04.674358 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.674338 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 14 01:35:04.674505 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.674479 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 14 01:35:04.674828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.674429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 14 01:35:04.675680 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.675660 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9c9j2"] Sep 14 01:35:04.675858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.675756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.678344 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.678323 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 14 01:35:04.678703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.678535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-s5dt8\"" Sep 14 01:35:04.678703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.678584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 14 01:35:04.678703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.678539 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:35:04.679904 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.679885 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99"] Sep 14 01:35:04.680042 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.680028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh"] Sep 14 01:35:04.680793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.680772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.681921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.681901 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44"] Sep 14 01:35:04.682658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.682639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9c9j2"] Sep 14 01:35:04.683712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.683692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-phmr4\"" Sep 14 01:35:04.683847 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.683723 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 14 01:35:04.683921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.683767 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:35:04.683921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.683786 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 14 01:35:04.684025 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.683789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 14 01:35:04.689074 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689214 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689280 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hp5\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 14 01:35:04.689495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.689634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.689515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd2039fe-5e89-469b-9995-a28af90bc447-serving-cert\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48j27\" (UniqueName: \"kubernetes.io/projected/fd2039fe-5e89-469b-9995-a28af90bc447-kube-api-access-48j27\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1edd9088-2b01-469a-abd3-e05031c14dee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hp5\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gq9\" (UniqueName: \"kubernetes.io/projected/345501e9-3b2a-402c-bfed-15cca6f5d180-kube-api-access-z5gq9\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2qq\" (UniqueName: \"kubernetes.io/projected/1edd9088-2b01-469a-abd3-e05031c14dee-kube-api-access-lr2qq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.790220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1edd9088-2b01-469a-abd3-e05031c14dee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-config\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:04.790383 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:04.790404 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb46c8f97-mzdn5: secret "image-registry-tls" not found Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:04.790477 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls podName:fe257aad-79a3-4190-b3ea-7e79c576e610 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:05.290458354 +0000 UTC m=+250.523823593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls") pod "image-registry-cb46c8f97-mzdn5" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610") : secret "image-registry-tls" not found Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-trusted-ca\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.790998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.790998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.790665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gmq\" (UniqueName: \"kubernetes.io/projected/a5392461-ac66-46c3-96b3-dc17eb4dac87-kube-api-access-q5gmq\") pod \"volume-data-source-validator-86695b7cc7-n4f99\" (UID: \"a5392461-ac66-46c3-96b3-dc17eb4dac87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" Sep 14 01:35:04.791133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.791114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.791186 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.791167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.791479 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.791463 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.792696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.792676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.792815 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.792798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.799122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.799097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.799266 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.799248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hp5\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:04.825146 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.825079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:04.891270 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-config\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.891270 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-trusted-ca\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gmq\" (UniqueName: \"kubernetes.io/projected/a5392461-ac66-46c3-96b3-dc17eb4dac87-kube-api-access-q5gmq\") pod \"volume-data-source-validator-86695b7cc7-n4f99\" (UID: \"a5392461-ac66-46c3-96b3-dc17eb4dac87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd2039fe-5e89-469b-9995-a28af90bc447-serving-cert\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48j27\" (UniqueName: \"kubernetes.io/projected/fd2039fe-5e89-469b-9995-a28af90bc447-kube-api-access-48j27\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1edd9088-2b01-469a-abd3-e05031c14dee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gq9\" (UniqueName: \"kubernetes.io/projected/345501e9-3b2a-402c-bfed-15cca6f5d180-kube-api-access-z5gq9\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2qq\" (UniqueName: \"kubernetes.io/projected/1edd9088-2b01-469a-abd3-e05031c14dee-kube-api-access-lr2qq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.891471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.891462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1edd9088-2b01-469a-abd3-e05031c14dee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.891839 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:04.891606 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 01:35:04.891839 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:04.891690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls podName:345501e9-3b2a-402c-bfed-15cca6f5d180 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:05.391667487 +0000 UTC m=+250.625032737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-htj44" (UID: "345501e9-3b2a-402c-bfed-15cca6f5d180") : secret "samples-operator-tls" not found Sep 14 01:35:04.892170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.892143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-config\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.892340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.892202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1edd9088-2b01-469a-abd3-e05031c14dee-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.892781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.892763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd2039fe-5e89-469b-9995-a28af90bc447-trusted-ca\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.893776 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.893756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1edd9088-2b01-469a-abd3-e05031c14dee-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.893898 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.893879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd2039fe-5e89-469b-9995-a28af90bc447-serving-cert\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.900119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.900092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gmq\" (UniqueName: \"kubernetes.io/projected/a5392461-ac66-46c3-96b3-dc17eb4dac87-kube-api-access-q5gmq\") pod \"volume-data-source-validator-86695b7cc7-n4f99\" (UID: \"a5392461-ac66-46c3-96b3-dc17eb4dac87\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" Sep 14 01:35:04.900296 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.900273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2qq\" (UniqueName: \"kubernetes.io/projected/1edd9088-2b01-469a-abd3-e05031c14dee-kube-api-access-lr2qq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tqpjh\" (UID: \"1edd9088-2b01-469a-abd3-e05031c14dee\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.900395 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.900378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gq9\" (UniqueName: \"kubernetes.io/projected/345501e9-3b2a-402c-bfed-15cca6f5d180-kube-api-access-z5gq9\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:04.900901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.900880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48j27\" (UniqueName: \"kubernetes.io/projected/fd2039fe-5e89-469b-9995-a28af90bc447-kube-api-access-48j27\") pod \"console-operator-696884fb9c-9c9j2\" (UID: \"fd2039fe-5e89-469b-9995-a28af90bc447\") " pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:04.912976 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.912951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:04.978712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.978682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" Sep 14 01:35:04.986970 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.986955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" Sep 14 01:35:04.997941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:04.997920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:05.252024 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.251992 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99"] Sep 14 01:35:05.255625 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:05.255601 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5392461_ac66_46c3_96b3_dc17eb4dac87.slice/crio-1f4982f0ca486f759d17e28008e871725a84931cc2e073bc18638b44f22577c8 WatchSource:0}: Error finding container 1f4982f0ca486f759d17e28008e871725a84931cc2e073bc18638b44f22577c8: Status 404 returned error can't find the container with id 1f4982f0ca486f759d17e28008e871725a84931cc2e073bc18638b44f22577c8 Sep 14 01:35:05.272356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.272332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh"] Sep 14 01:35:05.275492 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:05.275468 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd9088_2b01_469a_abd3_e05031c14dee.slice/crio-0063c90a61d035fdb7caa0a5d821f0db3c57d4087fdf1d5fabfa1274e679a0ac WatchSource:0}: Error finding container 0063c90a61d035fdb7caa0a5d821f0db3c57d4087fdf1d5fabfa1274e679a0ac: Status 404 returned error can't find the container with id 0063c90a61d035fdb7caa0a5d821f0db3c57d4087fdf1d5fabfa1274e679a0ac Sep 14 01:35:05.275911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.275889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-9c9j2"] Sep 14 01:35:05.277711 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:05.277683 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2039fe_5e89_469b_9995_a28af90bc447.slice/crio-94dab368504b93812392c6eb1b0a55106f6ae93ebf3f553e71450c87c8983ec3 WatchSource:0}: Error finding container 94dab368504b93812392c6eb1b0a55106f6ae93ebf3f553e71450c87c8983ec3: Status 404 returned error can't find the container with id 94dab368504b93812392c6eb1b0a55106f6ae93ebf3f553e71450c87c8983ec3 Sep 14 01:35:05.294682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.294656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:05.294809 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:05.294793 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 01:35:05.294858 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:05.294812 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb46c8f97-mzdn5: secret "image-registry-tls" not found Sep 14 01:35:05.294858 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:05.294854 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls podName:fe257aad-79a3-4190-b3ea-7e79c576e610 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:06.294840672 +0000 UTC m=+251.528205924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls") pod "image-registry-cb46c8f97-mzdn5" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610") : secret "image-registry-tls" not found Sep 14 01:35:05.395533 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.395440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:05.395679 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:05.395572 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 01:35:05.395679 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:05.395634 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls podName:345501e9-3b2a-402c-bfed-15cca6f5d180 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:06.395615578 +0000 UTC m=+251.628980828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-htj44" (UID: "345501e9-3b2a-402c-bfed-15cca6f5d180") : secret "samples-operator-tls" not found Sep 14 01:35:05.939150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.939114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" event={"ID":"fd2039fe-5e89-469b-9995-a28af90bc447","Type":"ContainerStarted","Data":"94dab368504b93812392c6eb1b0a55106f6ae93ebf3f553e71450c87c8983ec3"} Sep 14 01:35:05.940899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.940840 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" event={"ID":"a5392461-ac66-46c3-96b3-dc17eb4dac87","Type":"ContainerStarted","Data":"1f4982f0ca486f759d17e28008e871725a84931cc2e073bc18638b44f22577c8"} Sep 14 01:35:05.942468 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:05.942411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" event={"ID":"1edd9088-2b01-469a-abd3-e05031c14dee","Type":"ContainerStarted","Data":"0063c90a61d035fdb7caa0a5d821f0db3c57d4087fdf1d5fabfa1274e679a0ac"} Sep 14 01:35:06.303622 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:06.303578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:06.303818 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:06.303753 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 01:35:06.303818 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:06.303780 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb46c8f97-mzdn5: secret "image-registry-tls" not found Sep 14 01:35:06.304101 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:06.303842 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls podName:fe257aad-79a3-4190-b3ea-7e79c576e610 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:08.303826788 +0000 UTC m=+253.537192028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls") pod "image-registry-cb46c8f97-mzdn5" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610") : secret "image-registry-tls" not found Sep 14 01:35:06.404999 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:06.404954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:06.405193 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:06.405139 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 01:35:06.405261 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:06.405217 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls podName:345501e9-3b2a-402c-bfed-15cca6f5d180 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:08.405196051 +0000 UTC m=+253.638561286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-htj44" (UID: "345501e9-3b2a-402c-bfed-15cca6f5d180") : secret "samples-operator-tls" not found Sep 14 01:35:07.111044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.110998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:35:07.111524 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:07.111178 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:35:07.111524 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:07.111263 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs podName:d5450a90-678e-4192-ab47-81dcef253c6c nodeName:}" failed. No retries permitted until 2026-09-14 01:37:09.111240466 +0000 UTC m=+374.344605701 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs") pod "network-metrics-daemon-xhtx9" (UID: "d5450a90-678e-4192-ab47-81dcef253c6c") : secret "metrics-daemon-secret" not found Sep 14 01:35:07.951194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.951117 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/0.log" Sep 14 01:35:07.951194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.951158 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd2039fe-5e89-469b-9995-a28af90bc447" containerID="496eb9c4b14eda703d0c893497d606a4b75391d1daf2cda3ef72ef5cf835534d" exitCode=255 Sep 14 01:35:07.951387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.951229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" event={"ID":"fd2039fe-5e89-469b-9995-a28af90bc447","Type":"ContainerDied","Data":"496eb9c4b14eda703d0c893497d606a4b75391d1daf2cda3ef72ef5cf835534d"} Sep 14 01:35:07.951501 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.951484 2579 scope.go:117] "RemoveContainer" containerID="496eb9c4b14eda703d0c893497d606a4b75391d1daf2cda3ef72ef5cf835534d" Sep 14 01:35:07.953033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.953006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" event={"ID":"a5392461-ac66-46c3-96b3-dc17eb4dac87","Type":"ContainerStarted","Data":"ebe6df0689e8ca230a9f48fc19ca512c402bb7302591db8a343d0e9cc9c899de"} Sep 14 01:35:07.954582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.954565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" event={"ID":"1edd9088-2b01-469a-abd3-e05031c14dee","Type":"ContainerStarted","Data":"61c717fa4db49f20defd03b02e17f4b268f08a4faffe9d09328d66bbe7154e66"} Sep 14 01:35:07.977918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.977875 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-n4f99" podStartSLOduration=1.5886059129999999 podStartE2EDuration="3.977859859s" totalImagesPullingTime="2.389253946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:07.976979154 +0000 UTC m=+253.210344410" watchObservedRunningTime="2026-09-14 01:35:07.977859859 +0000 UTC m=+253.211225158" Sep 14 01:35:07.992552 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:07.992513 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" podStartSLOduration=1.614088105 podStartE2EDuration="3.992502237s" totalImagesPullingTime="2.378414132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:07.99158154 +0000 UTC m=+253.224946796" watchObservedRunningTime="2026-09-14 01:35:07.992502237 +0000 UTC m=+253.225867494" Sep 14 01:35:08.055621 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.055584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.159411 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.159374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.318785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.318751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:08.318978 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.318862 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 01:35:08.318978 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.318873 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb46c8f97-mzdn5: secret "image-registry-tls" not found Sep 14 01:35:08.318978 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.318919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls podName:fe257aad-79a3-4190-b3ea-7e79c576e610 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.318906433 +0000 UTC m=+257.552271673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls") pod "image-registry-cb46c8f97-mzdn5" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610") : secret "image-registry-tls" not found Sep 14 01:35:08.413147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.413120 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-64vfb"] Sep 14 01:35:08.416309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.416288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419103 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:08.419223 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.419219 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 01:35:08.419323 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419323 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4006e721-807a-4462-a3a0-ec6da9413fd4-data-volume\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419323 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.419267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls podName:345501e9-3b2a-402c-bfed-15cca6f5d180 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.419250363 +0000 UTC m=+257.652615597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-htj44" (UID: "345501e9-3b2a-402c-bfed-15cca6f5d180") : secret "samples-operator-tls" not found Sep 14 01:35:08.419485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8j7s\" (UniqueName: \"kubernetes.io/projected/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-api-access-f8j7s\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4006e721-807a-4462-a3a0-ec6da9413fd4-crio-socket\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.419583 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 01:35:08.419583 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419543 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 01:35:08.419772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.419757 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 01:35:08.424192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.424172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dqk8k\"" Sep 14 01:35:08.425128 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.425111 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 01:35:08.438097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.438073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.438225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.438175 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-64vfb"] Sep 14 01:35:08.519911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.519877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4006e721-807a-4462-a3a0-ec6da9413fd4-crio-socket\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.519934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.519991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4006e721-807a-4462-a3a0-ec6da9413fd4-crio-socket\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.520015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.520039 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.520043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4006e721-807a-4462-a3a0-ec6da9413fd4-data-volume\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520109 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.520105 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:09.020085997 +0000 UTC m=+254.253451231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:08.520463 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.520123 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:08.520463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.520152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8j7s\" (UniqueName: \"kubernetes.io/projected/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-api-access-f8j7s\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.520463 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.520171 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:09.020155044 +0000 UTC m=+254.253520279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : configmap "kube-rbac-proxy" not found Sep 14 01:35:08.520463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.520374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4006e721-807a-4462-a3a0-ec6da9413fd4-data-volume\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.530959 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.530933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8j7s\" (UniqueName: \"kubernetes.io/projected/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-api-access-f8j7s\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:08.669481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.669391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.733806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.733774 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd"] Sep 14 01:35:08.736661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.736645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" Sep 14 01:35:08.738992 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.738961 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 14 01:35:08.739117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.739081 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 14 01:35:08.739239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.739213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tst9q\"" Sep 14 01:35:08.744102 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.744023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd"] Sep 14 01:35:08.747933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.747909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.805118 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.805085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.822550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.822475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqdt\" (UniqueName: \"kubernetes.io/projected/12c4c6ea-daca-412b-afd5-981960e122e5-kube-api-access-skqdt\") pod \"migrator-8b4d697c5-k55dd\" (UID: \"12c4c6ea-daca-412b-afd5-981960e122e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" Sep 14 01:35:08.905738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.905697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:08.926880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.926817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqdt\" (UniqueName: \"kubernetes.io/projected/12c4c6ea-daca-412b-afd5-981960e122e5-kube-api-access-skqdt\") pod \"migrator-8b4d697c5-k55dd\" (UID: \"12c4c6ea-daca-412b-afd5-981960e122e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" Sep 14 01:35:08.934259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.934239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqdt\" (UniqueName: \"kubernetes.io/projected/12c4c6ea-daca-412b-afd5-981960e122e5-kube-api-access-skqdt\") pod \"migrator-8b4d697c5-k55dd\" (UID: \"12c4c6ea-daca-412b-afd5-981960e122e5\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" Sep 14 01:35:08.959383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.959364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:35:08.959733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.959720 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/0.log" Sep 14 01:35:08.959796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.959751 2579 generic.go:356] "Generic (PLEG): container finished" podID="fd2039fe-5e89-469b-9995-a28af90bc447" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" exitCode=255 Sep 14 01:35:08.959844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.959818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" event={"ID":"fd2039fe-5e89-469b-9995-a28af90bc447","Type":"ContainerDied","Data":"cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0"} Sep 14 01:35:08.959901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.959890 2579 scope.go:117] "RemoveContainer" containerID="496eb9c4b14eda703d0c893497d606a4b75391d1daf2cda3ef72ef5cf835534d" Sep 14 01:35:08.960110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:08.960088 2579 scope.go:117] "RemoveContainer" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" Sep 14 01:35:08.960331 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:08.960309 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9c9j2_openshift-console-operator(fd2039fe-5e89-469b-9995-a28af90bc447)\"" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" podUID="fd2039fe-5e89-469b-9995-a28af90bc447" Sep 14 01:35:09.017632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.017608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:09.027838 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.027819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:09.027979 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:09.027961 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:09.028015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.027976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:09.028049 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:09.028024 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:10.028007518 +0000 UTC m=+255.261372770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:09.028306 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:09.028287 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:09.028366 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:09.028353 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:10.028334624 +0000 UTC m=+255.261699874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : configmap "kube-rbac-proxy" not found Sep 14 01:35:09.048554 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.048537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" Sep 14 01:35:09.119172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.119137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:09.302117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.302089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:09.312616 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.312593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd"] Sep 14 01:35:09.315782 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:09.315764 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c4c6ea_daca_412b_afd5_981960e122e5.slice/crio-148e596de908a413fb5cde8a525c6ef94adfee03948533218605b3845a8906be WatchSource:0}: Error finding container 148e596de908a413fb5cde8a525c6ef94adfee03948533218605b3845a8906be: Status 404 returned error can't find the container with id 148e596de908a413fb5cde8a525c6ef94adfee03948533218605b3845a8906be Sep 14 01:35:09.402362 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.402310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:09.964299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.964273 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:35:09.964736 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.964652 2579 scope.go:117] "RemoveContainer" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" Sep 14 01:35:09.964876 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:09.964834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9c9j2_openshift-console-operator(fd2039fe-5e89-469b-9995-a28af90bc447)\"" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" podUID="fd2039fe-5e89-469b-9995-a28af90bc447" Sep 14 01:35:09.965899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:09.965870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" event={"ID":"12c4c6ea-daca-412b-afd5-981960e122e5","Type":"ContainerStarted","Data":"148e596de908a413fb5cde8a525c6ef94adfee03948533218605b3845a8906be"} Sep 14 01:35:10.035817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.035780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:10.035978 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:10.035962 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:10.036034 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:10.036023 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.036006747 +0000 UTC m=+257.269371982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : configmap "kube-rbac-proxy" not found Sep 14 01:35:10.036100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.036085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:10.036277 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:10.036256 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:10.036337 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:10.036321 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.036303685 +0000 UTC m=+257.269668930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:10.565904 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.565879 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szkpn_1006b6cf-ffe6-49bb-afdd-c86e5dd32791/dns-node-resolver/0.log" Sep 14 01:35:10.670047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.669959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:10.722008 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.721979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:10.926421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.926329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:10.971813 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.971784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" event={"ID":"12c4c6ea-daca-412b-afd5-981960e122e5","Type":"ContainerStarted","Data":"740e2a7fc8105fea33c7a819d15870148e3f59779125e75322d55631dae90fcb"} Sep 14 01:35:10.972070 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.971843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" event={"ID":"12c4c6ea-daca-412b-afd5-981960e122e5","Type":"ContainerStarted","Data":"f36aa18f5d6432384901bb4579fad7c5ad739b3164cc6666feddaced5e718ad7"} Sep 14 01:35:10.973239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.973212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:10.986875 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:10.986831 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-k55dd" podStartSLOduration=1.9180557710000001 podStartE2EDuration="2.986817025s" totalImagesPullingTime="1.068761254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:10.98544479 +0000 UTC m=+256.218810067" watchObservedRunningTime="2026-09-14 01:35:10.986817025 +0000 UTC m=+256.220182284" Sep 14 01:35:11.055781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:11.055754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:11.156426 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:11.156399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:11.165114 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:11.165096 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvncq_5eac84e6-0163-4182-a8ab-63e166b09d6f/node-ca/0.log" Sep 14 01:35:11.888216 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:11.888181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:12.048585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.048551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:12.048750 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.048615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:12.048750 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.048681 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:12.048750 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.048742 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:16.048726709 +0000 UTC m=+261.282091948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : configmap "kube-rbac-proxy" not found Sep 14 01:35:12.048857 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.048686 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:12.048857 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.048813 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:16.048801031 +0000 UTC m=+261.282166279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:12.168239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.168161 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n646v_6153521d-a4f5-4e75-9ce8-c477938c92d5/serve-healthcheck-canary/0.log" Sep 14 01:35:12.350779 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.350724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:12.350952 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.350899 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 01:35:12.350952 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.350922 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb46c8f97-mzdn5: secret "image-registry-tls" not found Sep 14 01:35:12.351024 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.350992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls podName:fe257aad-79a3-4190-b3ea-7e79c576e610 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:20.350974404 +0000 UTC m=+265.584339640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls") pod "image-registry-cb46c8f97-mzdn5" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610") : secret "image-registry-tls" not found Sep 14 01:35:12.451320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.451225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:12.451463 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.451398 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 01:35:12.451501 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:12.451477 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls podName:345501e9-3b2a-402c-bfed-15cca6f5d180 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:20.451460548 +0000 UTC m=+265.684825788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-htj44" (UID: "345501e9-3b2a-402c-bfed-15cca6f5d180") : secret "samples-operator-tls" not found Sep 14 01:35:12.789182 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:12.789152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:13.008330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:13.008290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:13.108645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:13.108551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:13.210126 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:13.210083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:13.310268 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:13.310238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:13.494014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:13.493986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:14.287604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.287566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:14.798550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.798517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:14.900309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.900219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:14.998254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.998222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:14.998409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.998310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:14.998587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:14.998575 2579 scope.go:117] "RemoveContainer" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" Sep 14 01:35:14.998783 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:14.998765 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9c9j2_openshift-console-operator(fd2039fe-5e89-469b-9995-a28af90bc447)\"" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" podUID="fd2039fe-5e89-469b-9995-a28af90bc447" Sep 14 01:35:15.084646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:15.084605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:15.301905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:15.301874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:15.493052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:15.493020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:15.990519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:15.990491 2579 scope.go:117] "RemoveContainer" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" Sep 14 01:35:15.990729 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:15.990642 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-9c9j2_openshift-console-operator(fd2039fe-5e89-469b-9995-a28af90bc447)\"" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" podUID="fd2039fe-5e89-469b-9995-a28af90bc447" Sep 14 01:35:16.077583 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.077554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:16.077748 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.077620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:16.077748 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:16.077688 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:16.077748 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:16.077714 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:16.077908 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:16.077755 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:24.077736136 +0000 UTC m=+269.311101374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:16.077908 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:16.077775 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm podName:4006e721-807a-4462-a3a0-ec6da9413fd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:24.077765887 +0000 UTC m=+269.311131126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-64vfb" (UID: "4006e721-807a-4462-a3a0-ec6da9413fd4") : configmap "kube-rbac-proxy" not found Sep 14 01:35:16.387045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.386958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:16.490263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.490229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:16.685022 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.684951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:16.731948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:16.731921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:17.092185 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:17.092153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:17.288839 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:17.288803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:17.885660 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:17.885630 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:18.127589 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:18.127511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:18.290647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:18.290620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:18.688489 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:18.688406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:18.884749 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:18.884720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:19.276974 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:19.276945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:19.486083 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:19.486032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:19.890458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:19.890426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:20.285658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.285633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:20.408890 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.408850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:20.411235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.411216 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"image-registry-cb46c8f97-mzdn5\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:20.486052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.486022 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:20.490498 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.490451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:20.509663 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.509627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:20.512003 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.511977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/345501e9-3b2a-402c-bfed-15cca6f5d180-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-htj44\" (UID: \"345501e9-3b2a-402c-bfed-15cca6f5d180\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:20.592985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.592951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" Sep 14 01:35:20.756257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.756216 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:20.759229 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:20.759204 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe257aad_79a3_4190_b3ea_7e79c576e610.slice/crio-aaf20794818fe3d674797e5c3e9962f05b486d7c937f38491d6e5cd666c14b40 WatchSource:0}: Error finding container aaf20794818fe3d674797e5c3e9962f05b486d7c937f38491d6e5cd666c14b40: Status 404 returned error can't find the container with id aaf20794818fe3d674797e5c3e9962f05b486d7c937f38491d6e5cd666c14b40 Sep 14 01:35:20.858654 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.858627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44"] Sep 14 01:35:20.887936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:20.887909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:21.007689 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.007653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" event={"ID":"345501e9-3b2a-402c-bfed-15cca6f5d180","Type":"ContainerStarted","Data":"269f3008a4b3e877379b78c1dc8c784557ea02409b848d4a7a7fdb63a8826ff9"} Sep 14 01:35:21.009048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.009020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" event={"ID":"fe257aad-79a3-4190-b3ea-7e79c576e610","Type":"ContainerStarted","Data":"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59"} Sep 14 01:35:21.009173 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.009070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" event={"ID":"fe257aad-79a3-4190-b3ea-7e79c576e610","Type":"ContainerStarted","Data":"aaf20794818fe3d674797e5c3e9962f05b486d7c937f38491d6e5cd666c14b40"} Sep 14 01:35:21.009173 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.009165 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:21.026858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.026803 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" podStartSLOduration=17.026783408 podStartE2EDuration="17.026783408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:21.025778831 +0000 UTC m=+266.259144111" watchObservedRunningTime="2026-09-14 01:35:21.026783408 +0000 UTC m=+266.260148668" Sep 14 01:35:21.667300 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.667259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:21.768235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.768193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:21.888509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:21.888461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:22.290797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:22.290744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:22.489184 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:22.489149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:22.833392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:22.833323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:22.886209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:22.886178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:23.016248 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.016213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" event={"ID":"345501e9-3b2a-402c-bfed-15cca6f5d180","Type":"ContainerStarted","Data":"aa77344fb4744d58332259021f901f7b1a117b735a206ed747d9df05fb8d08ec"} Sep 14 01:35:23.016436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.016306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" event={"ID":"345501e9-3b2a-402c-bfed-15cca6f5d180","Type":"ContainerStarted","Data":"82b48c9f442d65e2fe9d7a92ffdf0269d464af285a23c64e0fb4e6c2596530df"} Sep 14 01:35:23.031160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.031114 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-htj44" podStartSLOduration=17.373763152 podStartE2EDuration="19.031099533s" totalImagesPullingTime="1.657336381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:23.030425293 +0000 UTC m=+268.263790550" watchObservedRunningTime="2026-09-14 01:35:23.031099533 +0000 UTC m=+268.264464789" Sep 14 01:35:23.291513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.291477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:23.702127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.702019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:23.898036 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:23.897999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:24.139326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.139289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:24.139527 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.139366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:24.139816 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.139797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4006e721-807a-4462-a3a0-ec6da9413fd4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:24.141600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.141580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4006e721-807a-4462-a3a0-ec6da9413fd4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-64vfb\" (UID: \"4006e721-807a-4462-a3a0-ec6da9413fd4\") " pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:24.325806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.325772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-64vfb" Sep 14 01:35:24.593741 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:24.593701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-64vfb"] Sep 14 01:35:24.596678 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:24.596650 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4006e721_807a_4462_a3a0_ec6da9413fd4.slice/crio-5d3631c9b8d354c70d222e130d56925290ec4d50ff3f3ddfd79b637e2a18d58d WatchSource:0}: Error finding container 5d3631c9b8d354c70d222e130d56925290ec4d50ff3f3ddfd79b637e2a18d58d: Status 404 returned error can't find the container with id 5d3631c9b8d354c70d222e130d56925290ec4d50ff3f3ddfd79b637e2a18d58d Sep 14 01:35:25.024160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:25.024128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64vfb" event={"ID":"4006e721-807a-4462-a3a0-ec6da9413fd4","Type":"ContainerStarted","Data":"a5effbaf0d2faa4bc0b7a4993d1aaff8ce67437e3466d8b106cc3ae7a860518c"} Sep 14 01:35:25.024564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:25.024184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64vfb" event={"ID":"4006e721-807a-4462-a3a0-ec6da9413fd4","Type":"ContainerStarted","Data":"5d3631c9b8d354c70d222e130d56925290ec4d50ff3f3ddfd79b637e2a18d58d"} Sep 14 01:35:26.029449 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:26.029416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64vfb" event={"ID":"4006e721-807a-4462-a3a0-ec6da9413fd4","Type":"ContainerStarted","Data":"b3da1af36e234c11fb5e9255e297bab3ac14552f7ae6120c24ee1fbe42d5b6a1"} Sep 14 01:35:27.034652 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:27.034619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-64vfb" event={"ID":"4006e721-807a-4462-a3a0-ec6da9413fd4","Type":"ContainerStarted","Data":"528a148f832c00cd190a69a63c9bd133b97f8506b7617f83eeec8e1ba24a7b76"} Sep 14 01:35:27.051564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:27.051519 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-64vfb" podStartSLOduration=17.206220339 podStartE2EDuration="19.051504595s" totalImagesPullingTime="1.845284256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:35:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:27.051471764 +0000 UTC m=+272.284837040" watchObservedRunningTime="2026-09-14 01:35:27.051504595 +0000 UTC m=+272.284869847" Sep 14 01:35:28.128239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:28.128202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:28.182072 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:28.182028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:29.203885 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.203856 2579 scope.go:117] "RemoveContainer" containerID="cae67ad6899ba100b8644bc749222bfafb9bcec2a5bc0e8059e36ce9f6c8a1b0" Sep 14 01:35:29.832417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.832387 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:29.850032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.850003 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl"] Sep 14 01:35:29.853180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.853164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:29.855871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.855853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 14 01:35:29.856099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.856084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gjmjl\"" Sep 14 01:35:29.862906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.862882 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl"] Sep 14 01:35:29.882162 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.882138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/20670962-98ca-454a-a115-638fce2ade07-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rmnzl\" (UID: \"20670962-98ca-454a-a115-638fce2ade07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:29.885072 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.885040 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bb7f9cb9c-fzspk"] Sep 14 01:35:29.887898 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.887884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.899791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.899769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb7f9cb9c-fzspk"] Sep 14 01:35:29.961636 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.961601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:29.983415 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3d93b1f-85ea-4e71-8220-c90b277f74d5-ca-trust-extracted\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-bound-sa-token\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/20670962-98ca-454a-a115-638fce2ade07-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rmnzl\" (UID: \"20670962-98ca-454a-a115-638fce2ade07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:29.983536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-trusted-ca\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-image-registry-private-configuration\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-installation-pull-secrets\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwk2b\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-kube-api-access-nwk2b\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-tls\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.983800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.983681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-certificates\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:29.985963 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:29.985935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/20670962-98ca-454a-a115-638fce2ade07-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rmnzl\" (UID: \"20670962-98ca-454a-a115-638fce2ade07\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:30.047195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.047166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:35:30.047330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.047261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" event={"ID":"fd2039fe-5e89-469b-9995-a28af90bc447","Type":"ContainerStarted","Data":"8bd244ce601b5cf2fb4434fe3f4979b28923be56a29a5c1123123e0c2f10d874"} Sep 14 01:35:30.047548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.047527 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:30.053610 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.053559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" Sep 14 01:35:30.070254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.070199 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-9c9j2" podStartSLOduration=23.696356861 podStartE2EDuration="26.0701816s" totalImagesPullingTime="2.373824739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:30.069219646 +0000 UTC m=+275.302584904" watchObservedRunningTime="2026-09-14 01:35:30.0701816 +0000 UTC m=+275.303546858" Sep 14 01:35:30.072320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.072300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:30.084489 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-trusted-ca\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.084594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-image-registry-private-configuration\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.084594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-installation-pull-secrets\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.084700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwk2b\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-kube-api-access-nwk2b\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.084700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-tls\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.084700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.084691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-certificates\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.086496 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.086470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-trusted-ca\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.086719 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.086692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3d93b1f-85ea-4e71-8220-c90b277f74d5-ca-trust-extracted\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.086803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.086785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-bound-sa-token\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.087100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.086800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-certificates\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.090782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.087828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-image-registry-private-configuration\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.090782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.088215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e3d93b1f-85ea-4e71-8220-c90b277f74d5-ca-trust-extracted\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.090782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.088640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e3d93b1f-85ea-4e71-8220-c90b277f74d5-installation-pull-secrets\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.091930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.091905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-registry-tls\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.103867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.103848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwk2b\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-kube-api-access-nwk2b\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.103971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.103920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3d93b1f-85ea-4e71-8220-c90b277f74d5-bound-sa-token\") pod \"image-registry-6bb7f9cb9c-fzspk\" (UID: \"e3d93b1f-85ea-4e71-8220-c90b277f74d5\") " pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.160851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.160810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:30.162252 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.162236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:30.198559 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.198529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:30.259956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.259868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:30.393961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.393888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:30.426393 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.426365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl"] Sep 14 01:35:30.429578 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:30.429560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20670962_98ca_454a_a115_638fce2ade07.slice/crio-18ba0a8c0d2d140ef446db2d5e611ea49723f4b42512adb4c692da50a1700c72 WatchSource:0}: Error finding container 18ba0a8c0d2d140ef446db2d5e611ea49723f4b42512adb4c692da50a1700c72: Status 404 returned error can't find the container with id 18ba0a8c0d2d140ef446db2d5e611ea49723f4b42512adb4c692da50a1700c72 Sep 14 01:35:30.475699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.475678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bb7f9cb9c-fzspk"] Sep 14 01:35:30.477892 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:30.477863 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d93b1f_85ea_4e71_8220_c90b277f74d5.slice/crio-0f26704351349c2ccbd3f79cbdcd13c8dfb864a8600d19435ac488f1c339efcc WatchSource:0}: Error finding container 0f26704351349c2ccbd3f79cbdcd13c8dfb864a8600d19435ac488f1c339efcc: Status 404 returned error can't find the container with id 0f26704351349c2ccbd3f79cbdcd13c8dfb864a8600d19435ac488f1c339efcc Sep 14 01:35:30.766326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.766287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:30.821973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:30.821933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:31.052298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.052219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" event={"ID":"e3d93b1f-85ea-4e71-8220-c90b277f74d5","Type":"ContainerStarted","Data":"869bd8eec085f221b11e65e97026e7d0ac05be4ba6841e3b4a730e2d12f79c40"} Sep 14 01:35:31.052298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.052275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" event={"ID":"e3d93b1f-85ea-4e71-8220-c90b277f74d5","Type":"ContainerStarted","Data":"0f26704351349c2ccbd3f79cbdcd13c8dfb864a8600d19435ac488f1c339efcc"} Sep 14 01:35:31.052487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.052336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:31.053327 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.053307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" event={"ID":"20670962-98ca-454a-a115-638fce2ade07","Type":"ContainerStarted","Data":"18ba0a8c0d2d140ef446db2d5e611ea49723f4b42512adb4c692da50a1700c72"} Sep 14 01:35:31.070223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.070179 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" podStartSLOduration=2.070166663 podStartE2EDuration="2.070166663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:31.069359095 +0000 UTC m=+276.302724372" watchObservedRunningTime="2026-09-14 01:35:31.070166663 +0000 UTC m=+276.303531919" Sep 14 01:35:31.167225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.167182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:31.268247 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.268206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:31.383366 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.383336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:31.575895 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.575862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:31.977508 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:31.977476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:32.033409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.033373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:32.058510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.058478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" event={"ID":"20670962-98ca-454a-a115-638fce2ade07","Type":"ContainerStarted","Data":"fc1219bf6bed1d84b4ba37cdb147fbe866b22212591717040b46a4d89f03c53c"} Sep 14 01:35:32.058816 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.058795 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:32.063356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.063337 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" Sep 14 01:35:32.072691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.072651 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rmnzl" podStartSLOduration=2.155702428 podStartE2EDuration="3.072638664s" totalImagesPullingTime="916.936236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:32.07133612 +0000 UTC m=+277.304701396" watchObservedRunningTime="2026-09-14 01:35:32.072638664 +0000 UTC m=+277.306003922" Sep 14 01:35:32.917336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.917300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:32.978133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:32.978096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:33.180872 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:33.180789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:33.233048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:33.233019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:34.930783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:34.930746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:36.711225 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:36.711161 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4vzjs" podUID="685a7a90-19bb-49dc-b5df-54ae3937a55a" Sep 14 01:35:37.042505 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.042463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.075441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.075413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:37.143497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.143457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.267300 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.267270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.327637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.327562 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw"] Sep 14 01:35:37.330491 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.330470 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.332707 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.332683 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 01:35:37.332859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.332838 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 14 01:35:37.332988 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.332906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 01:35:37.332988 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.332928 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 01:35:37.332988 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.332957 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 14 01:35:37.334180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.334165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-6qqmp\"" Sep 14 01:35:37.334328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.334315 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 14 01:35:37.338586 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.338568 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2v2m"] Sep 14 01:35:37.340631 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.340615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.341278 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.341260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw"] Sep 14 01:35:37.343141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.343121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 01:35:37.343141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.343133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 01:35:37.343304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.343154 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wvnzr\"" Sep 14 01:35:37.343431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.343411 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 01:35:37.362223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.362197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.422447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.422409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.440208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkqf\" (UniqueName: \"kubernetes.io/projected/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-api-access-xnkqf\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttbp\" (UniqueName: \"kubernetes.io/projected/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-kube-api-access-pttbp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-metrics-client-ca\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-sys\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.440541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-textfile\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-wtmp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.440802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.440629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-root\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.523624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.523578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.541566 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-root\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.541696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.541696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-root\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.541793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.541793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.541880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkqf\" (UniqueName: \"kubernetes.io/projected/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-api-access-xnkqf\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.541880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttbp\" (UniqueName: \"kubernetes.io/projected/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-kube-api-access-pttbp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.541880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.542009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.541907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.542009 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:37.541995 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 01:35:37.542179 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:37.542086 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls podName:e7dd27df-a1ba-40c0-ac7e-2844aca312bc nodeName:}" failed. No retries permitted until 2026-09-14 01:35:38.042043745 +0000 UTC m=+283.275408981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls") pod "node-exporter-v2v2m" (UID: "e7dd27df-a1ba-40c0-ac7e-2844aca312bc") : secret "node-exporter-tls" not found Sep 14 01:35:37.542241 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542292 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-metrics-client-ca\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-sys\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.542400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.542510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-textfile\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-wtmp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-sys\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.542650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-wtmp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542824 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-textfile\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.542936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542871 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-metrics-client-ca\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.543030 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.542985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.544648 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.544625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.544853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.544830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.544895 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.544870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.552511 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.552483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttbp\" (UniqueName: \"kubernetes.io/projected/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-kube-api-access-pttbp\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:37.553199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.553183 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkqf\" (UniqueName: \"kubernetes.io/projected/b4c12dc5-82b4-40e7-a017-2aceb2e305ab-kube-api-access-xnkqf\") pod \"kube-state-metrics-59fbdd77d-cm5pw\" (UID: \"b4c12dc5-82b4-40e7-a017-2aceb2e305ab\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.623425 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.623340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.640240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.640218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" Sep 14 01:35:37.743863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.743827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:37.907516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:37.907373 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw"] Sep 14 01:35:37.909810 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:37.909786 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c12dc5_82b4_40e7_a017_2aceb2e305ab.slice/crio-4ddb1a92a55fda7015efa7fad59fa3e59654e0ae5d4977f727a103682a692723 WatchSource:0}: Error finding container 4ddb1a92a55fda7015efa7fad59fa3e59654e0ae5d4977f727a103682a692723: Status 404 returned error can't find the container with id 4ddb1a92a55fda7015efa7fad59fa3e59654e0ae5d4977f727a103682a692723 Sep 14 01:35:38.008128 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.008097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:38.046212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.046179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:38.048405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.048383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7dd27df-a1ba-40c0-ac7e-2844aca312bc-node-exporter-tls\") pod \"node-exporter-v2v2m\" (UID: \"e7dd27df-a1ba-40c0-ac7e-2844aca312bc\") " pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:38.080214 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.080186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" event={"ID":"b4c12dc5-82b4-40e7-a017-2aceb2e305ab","Type":"ContainerStarted","Data":"4ddb1a92a55fda7015efa7fad59fa3e59654e0ae5d4977f727a103682a692723"} Sep 14 01:35:38.105652 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.105621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:38.251370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.251344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2v2m" Sep 14 01:35:38.261208 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:38.261182 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7dd27df_a1ba_40c0_ac7e_2844aca312bc.slice/crio-f8d7604061e763060ecd6d21af23217c0c2b5b67ff23c5f7110db9313e1a5a28 WatchSource:0}: Error finding container f8d7604061e763060ecd6d21af23217c0c2b5b67ff23c5f7110db9313e1a5a28: Status 404 returned error can't find the container with id f8d7604061e763060ecd6d21af23217c0c2b5b67ff23c5f7110db9313e1a5a28 Sep 14 01:35:38.267950 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.267927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:38.338549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.338519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:38.343688 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.343666 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:35:38.346803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.346789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.349385 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349370 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 01:35:38.349466 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349384 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349743 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349750 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349762 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 01:35:38.349851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.349673 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sxcm9\"" Sep 14 01:35:38.359865 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.359845 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:35:38.449772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.449951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.449951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.449951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q582\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.449951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.449951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.449967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.450319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.450217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q582\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.551874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.551828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.552304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.552283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.552375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.552331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.552375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.552363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.552478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.552393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.553105 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.552679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.553563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.553365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.553563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.553517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.554808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.554783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.555367 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.555344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.555822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.555779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.555942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.555919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.556824 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.556797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.557150 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.557129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.558225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.558189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.558679 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.558662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.558939 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.558907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.560375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.560357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q582\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582\") pod \"alertmanager-main-0\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.586073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.586038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:38.656083 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.656043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:35:38.939841 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:38.939718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:35:39.085332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:39.085295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2v2m" event={"ID":"e7dd27df-a1ba-40c0-ac7e-2844aca312bc","Type":"ContainerStarted","Data":"f8d7604061e763060ecd6d21af23217c0c2b5b67ff23c5f7110db9313e1a5a28"} Sep 14 01:35:39.120606 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:39.120576 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495d8e9e_2c23_424f_a9bc_1b0714f3e8e2.slice/crio-3843a47af2c7256a438ed7765c8c4c6d2ef21fc141d9fc0d5e7c73c2374ba647 WatchSource:0}: Error finding container 3843a47af2c7256a438ed7765c8c4c6d2ef21fc141d9fc0d5e7c73c2374ba647: Status 404 returned error can't find the container with id 3843a47af2c7256a438ed7765c8c4c6d2ef21fc141d9fc0d5e7c73c2374ba647 Sep 14 01:35:39.838646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:39.838611 2579 patch_prober.go:28] interesting pod/image-registry-cb46c8f97-mzdn5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 14 01:35:39.838807 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:39.838664 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:35:40.091803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.091710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" event={"ID":"b4c12dc5-82b4-40e7-a017-2aceb2e305ab","Type":"ContainerStarted","Data":"a3eb8eb990214859147fd0ab63f94ce6991e9de8b17ae8734ef275e5b15e2671"} Sep 14 01:35:40.092262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.091822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" event={"ID":"b4c12dc5-82b4-40e7-a017-2aceb2e305ab","Type":"ContainerStarted","Data":"503aac795c170fde84b49040e4004d573157373cc48b56ab8561944d963c2998"} Sep 14 01:35:40.092262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.091861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" event={"ID":"b4c12dc5-82b4-40e7-a017-2aceb2e305ab","Type":"ContainerStarted","Data":"3c3b401111cecfaaa078c120c086a467337369408276ef9da09ce169daf4da27"} Sep 14 01:35:40.093584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.093556 2579 generic.go:356] "Generic (PLEG): container finished" podID="e7dd27df-a1ba-40c0-ac7e-2844aca312bc" containerID="a4107db782ffad292e9ce73b4a9b1e707ed0b224a8a7d319cd3aa89af70d83b9" exitCode=0 Sep 14 01:35:40.093717 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.093645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2v2m" event={"ID":"e7dd27df-a1ba-40c0-ac7e-2844aca312bc","Type":"ContainerDied","Data":"a4107db782ffad292e9ce73b4a9b1e707ed0b224a8a7d319cd3aa89af70d83b9"} Sep 14 01:35:40.095084 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.095041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"3843a47af2c7256a438ed7765c8c4c6d2ef21fc141d9fc0d5e7c73c2374ba647"} Sep 14 01:35:40.109794 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.109754 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-cm5pw" podStartSLOduration=1.6787268279999998 podStartE2EDuration="3.109736849s" totalImagesPullingTime="1.431010021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:40.10834365 +0000 UTC m=+285.341708935" watchObservedRunningTime="2026-09-14 01:35:40.109736849 +0000 UTC m=+285.343102087" Sep 14 01:35:40.166113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.166025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:40.168836 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.168810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/685a7a90-19bb-49dc-b5df-54ae3937a55a-metrics-tls\") pod \"dns-default-4vzjs\" (UID: \"685a7a90-19bb-49dc-b5df-54ae3937a55a\") " pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:40.211571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.211543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:40.311746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.311722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:40.378855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.378832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jfmw6\"" Sep 14 01:35:40.386784 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.386765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:40.544850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.544816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:40.651948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.651749 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vzjs"] Sep 14 01:35:40.654339 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:35:40.654313 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685a7a90_19bb_49dc_b5df_54ae3937a55a.slice/crio-c55f6e215615dc7fcc98a0fb462608398456744a2428e5350019726acf82eeec WatchSource:0}: Error finding container c55f6e215615dc7fcc98a0fb462608398456744a2428e5350019726acf82eeec: Status 404 returned error can't find the container with id c55f6e215615dc7fcc98a0fb462608398456744a2428e5350019726acf82eeec Sep 14 01:35:40.656284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:40.656214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:41.102878 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.102842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2v2m" event={"ID":"e7dd27df-a1ba-40c0-ac7e-2844aca312bc","Type":"ContainerStarted","Data":"7e8a33b5702c2e5cb86678e9259cadda30e0abdbb85a1a3b2234fa875d50f02d"} Sep 14 01:35:41.103336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.102909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2v2m" event={"ID":"e7dd27df-a1ba-40c0-ac7e-2844aca312bc","Type":"ContainerStarted","Data":"fffa428b08e4383fbfcee90c291e30f381ddef07c859343092959a246641f4a7"} Sep 14 01:35:41.104672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.104640 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4" exitCode=0 Sep 14 01:35:41.104801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.104731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4"} Sep 14 01:35:41.106358 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.106331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vzjs" event={"ID":"685a7a90-19bb-49dc-b5df-54ae3937a55a","Type":"ContainerStarted","Data":"c55f6e215615dc7fcc98a0fb462608398456744a2428e5350019726acf82eeec"} Sep 14 01:35:41.121458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:41.121416 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2v2m" podStartSLOduration=2.960843644 podStartE2EDuration="4.121402304s" totalImagesPullingTime="1.081880774s" totalInitContainerRuntime="78.677886ms" isStatefulPod=true podCreationTimestamp="2026-09-14 01:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:41.119465389 +0000 UTC m=+286.352830669" watchObservedRunningTime="2026-09-14 01:35:41.121402304 +0000 UTC m=+286.354767567" Sep 14 01:35:42.059543 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:42.059476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:42.150444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:42.150401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:42.560340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:42.560293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:43.115772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:43.115744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vzjs" event={"ID":"685a7a90-19bb-49dc-b5df-54ae3937a55a","Type":"ContainerStarted","Data":"78db00f488235611a1a3b46731cc935960476f1afe3ea92700d9a36ec108981c"} Sep 14 01:35:43.115873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:43.115838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vzjs" event={"ID":"685a7a90-19bb-49dc-b5df-54ae3937a55a","Type":"ContainerStarted","Data":"e463d954799f564810ac396109eb31fdfd5ca68eccbc08680276949fb96872c0"} Sep 14 01:35:43.115946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:43.115896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:43.470489 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:43.470455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:44.124117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.124078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd"} Sep 14 01:35:44.124288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.124148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0"} Sep 14 01:35:44.124288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.124175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553"} Sep 14 01:35:44.124288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.124204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e"} Sep 14 01:35:44.124288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.124237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7"} Sep 14 01:35:44.800535 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:44.800498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:45.131545 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:45.131454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerStarted","Data":"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5"} Sep 14 01:35:45.161298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:45.161247 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4vzjs" podStartSLOduration=253.583246821 podStartE2EDuration="4m15.161232259s" totalImagesPullingTime="1.577985438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:43.131640831 +0000 UTC m=+288.365006107" watchObservedRunningTime="2026-09-14 01:35:45.161232259 +0000 UTC m=+290.394597517" Sep 14 01:35:45.161460 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:45.161363 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4585797 podStartE2EDuration="7.161357787s" totalImagesPullingTime="3.686504345s" totalInitContainerRuntime="16.273742ms" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:45.155663938 +0000 UTC m=+290.389029196" watchObservedRunningTime="2026-09-14 01:35:45.161357787 +0000 UTC m=+290.394723044" Sep 14 01:35:45.954693 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:45.954663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:46.004343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:46.004317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:46.594785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:46.594747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:46.697477 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:46.697444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:47.121455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:47.121415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:47.167946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:47.167912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:49.036374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:49.036339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:49.836766 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:49.836736 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:50.204776 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:50.204698 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xhtx9" podUID="d5450a90-678e-4192-ab47-81dcef253c6c" Sep 14 01:35:50.278844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:50.278809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:50.377189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:50.377152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:50.480487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:50.480453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:51.157009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:51.156979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:51.256290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:51.256258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:51.605982 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:51.605946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:51.708053 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:51.708013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:52.064567 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:52.064538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bb7f9cb9c-fzspk" Sep 14 01:35:52.195427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:52.195391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:52.295732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:52.295696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:52.395814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:52.395732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:53.126136 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:53.126102 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4vzjs" Sep 14 01:35:54.851010 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:54.850944 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerName="registry" containerID="cri-o://970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59" gracePeriod=30 Sep 14 01:35:55.022591 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.022553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:55.074516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.074486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:55.087931 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.087909 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:55.135025 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.134955 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:35:55.135025 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.134963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:35:55.147427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.147407 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 01:35:55.168472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.168341 2579 generic.go:356] "Generic (PLEG): container finished" podID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerID="970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59" exitCode=0 Sep 14 01:35:55.168472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.168384 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" Sep 14 01:35:55.168472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.168392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" event={"ID":"fe257aad-79a3-4190-b3ea-7e79c576e610","Type":"ContainerDied","Data":"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59"} Sep 14 01:35:55.168472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.168463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb46c8f97-mzdn5" event={"ID":"fe257aad-79a3-4190-b3ea-7e79c576e610","Type":"ContainerDied","Data":"aaf20794818fe3d674797e5c3e9962f05b486d7c937f38491d6e5cd666c14b40"} Sep 14 01:35:55.185783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.168485 2579 scope.go:117] "RemoveContainer" containerID="970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59" Sep 14 01:35:55.185783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.176385 2579 scope.go:117] "RemoveContainer" containerID="970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59" Sep 14 01:35:55.185783 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:35:55.176634 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59\": container with ID starting with 970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59 not found: ID does not exist" containerID="970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59" Sep 14 01:35:55.185783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.176657 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59"} err="failed to get container status \"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59\": rpc error: code = NotFound desc = could not find container \"970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59\": container with ID starting with 970b932792ad9d02f847d1c0b9d8e01cb9ba6f30304042a041aa05d020b7da59 not found: ID does not exist" Sep 14 01:35:55.192085 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192050 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hp5\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192370 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration\") pod \"fe257aad-79a3-4190-b3ea-7e79c576e610\" (UID: \"fe257aad-79a3-4190-b3ea-7e79c576e610\") " Sep 14 01:35:55.192681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:35:55.192878 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.192854 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:35:55.194497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.194471 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:35:55.194590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.194550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:35:55.194815 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.194796 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:35:55.194868 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.194813 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:35:55.194968 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.194927 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5" (OuterVolumeSpecName: "kube-api-access-z6hp5") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "kube-api-access-z6hp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:35:55.200620 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.200600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fe257aad-79a3-4190-b3ea-7e79c576e610" (UID: "fe257aad-79a3-4190-b3ea-7e79c576e610"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:35:55.293789 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293760 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-trusted-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293789 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293783 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-bound-sa-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293789 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293791 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hp5\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-kube-api-access-z6hp5\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293800 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-certificates\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293809 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fe257aad-79a3-4190-b3ea-7e79c576e610-registry-tls\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293816 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-installation-pull-secrets\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293824 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fe257aad-79a3-4190-b3ea-7e79c576e610-image-registry-private-configuration\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.293984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.293833 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fe257aad-79a3-4190-b3ea-7e79c576e610-ca-trust-extracted\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:35:55.483265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.483234 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:55.486340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:55.486319 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb46c8f97-mzdn5"] Sep 14 01:35:57.208425 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:57.208386 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" path="/var/lib/kubelet/pods/fe257aad-79a3-4190-b3ea-7e79c576e610/volumes" Sep 14 01:35:58.129245 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:58.129214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:35:58.189742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:35:58.189709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:02.238703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:02.238669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:02.340744 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:02.340713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:04.203345 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:04.203308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:36:04.600232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:04.600189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:04.655173 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:04.655138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:07.212028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:07.211993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:07.312803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:07.312765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:07.413156 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:07.413126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:07.463240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:07.463167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:11.241933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:11.241902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:11.343072 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:11.343034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:13.130279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:13.130245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:13.229828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:13.229793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:43.354184 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:43.354148 2579 generic.go:356] "Generic (PLEG): container finished" podID="1edd9088-2b01-469a-abd3-e05031c14dee" containerID="61c717fa4db49f20defd03b02e17f4b268f08a4faffe9d09328d66bbe7154e66" exitCode=0 Sep 14 01:36:43.354560 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:43.354219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" event={"ID":"1edd9088-2b01-469a-abd3-e05031c14dee","Type":"ContainerDied","Data":"61c717fa4db49f20defd03b02e17f4b268f08a4faffe9d09328d66bbe7154e66"} Sep 14 01:36:43.354560 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:43.354544 2579 scope.go:117] "RemoveContainer" containerID="61c717fa4db49f20defd03b02e17f4b268f08a4faffe9d09328d66bbe7154e66" Sep 14 01:36:44.359827 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:44.359798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tqpjh" event={"ID":"1edd9088-2b01-469a-abd3-e05031c14dee","Type":"ContainerStarted","Data":"d8e3df264e00a3a6c2bf9c75ece3266b17729a37c1f4e533554567061dcb70e8"} Sep 14 01:36:53.354436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:53.354353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:53.406786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:53.406761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:57.419035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.418985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:57.472262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.472234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:57.595664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595628 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:36:57.595922 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595885 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="alertmanager" containerID="cri-o://45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" gracePeriod=120 Sep 14 01:36:57.595990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595960 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-metric" containerID="cri-o://e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" gracePeriod=120 Sep 14 01:36:57.596048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595998 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="config-reloader" containerID="cri-o://c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" gracePeriod=120 Sep 14 01:36:57.596048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595989 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy" containerID="cri-o://590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" gracePeriod=120 Sep 14 01:36:57.596048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.596016 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="prom-label-proxy" containerID="cri-o://35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" gracePeriod=120 Sep 14 01:36:57.596048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:57.595971 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-web" containerID="cri-o://3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" gracePeriod=120 Sep 14 01:36:58.419007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.418975 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" exitCode=0 Sep 14 01:36:58.419007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.418999 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" exitCode=0 Sep 14 01:36:58.419007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419005 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" exitCode=0 Sep 14 01:36:58.419007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419011 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" exitCode=0 Sep 14 01:36:58.419562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5"} Sep 14 01:36:58.419562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0"} Sep 14 01:36:58.419562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e"} Sep 14 01:36:58.419562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.419265 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7"} Sep 14 01:36:58.420724 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.420703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:58.477079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.477030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:58.563428 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.563374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:58.613668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.613636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:58.844723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:58.844697 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.000478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000485 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000502 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000708 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q582\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.000979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.000854 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db\") pod \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\" (UID: \"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2\") " Sep 14 01:36:59.001600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.001351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:36:59.001600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.001573 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:36:59.001979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.001948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:36:59.003690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.003651 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out" (OuterVolumeSpecName: "config-out") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:36:59.003690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.003650 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:36:59.003820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.003729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.004390 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.004353 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume" (OuterVolumeSpecName: "config-volume") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.004545 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.004517 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.004708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.004676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.004803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.004717 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582" (OuterVolumeSpecName: "kube-api-access-9q582") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "kube-api-access-9q582". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:36:59.005143 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.005120 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.008343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.008321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.014299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.014280 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config" (OuterVolumeSpecName: "web-config") pod "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" (UID: "495d8e9e-2c23-424f-a9bc-1b0714f3e8e2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:36:59.102153 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102111 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102153 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102152 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102153 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102167 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-main-tls\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102180 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102192 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102206 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102218 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q582\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-kube-api-access-9q582\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102230 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-cluster-tls-config\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102242 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-metrics-client-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102253 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-alertmanager-main-db\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102264 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-config-out\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102274 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-web-config\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.102354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.102284 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2-tls-assets\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:36:59.160417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.160383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:36:59.426386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426303 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" exitCode=0 Sep 14 01:36:59.426386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426328 2579 generic.go:356] "Generic (PLEG): container finished" podID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerID="3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" exitCode=0 Sep 14 01:36:59.426386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd"} Sep 14 01:36:59.426386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426373 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.426942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553"} Sep 14 01:36:59.426942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"495d8e9e-2c23-424f-a9bc-1b0714f3e8e2","Type":"ContainerDied","Data":"3843a47af2c7256a438ed7765c8c4c6d2ef21fc141d9fc0d5e7c73c2374ba647"} Sep 14 01:36:59.426942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.426500 2579 scope.go:117] "RemoveContainer" containerID="35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" Sep 14 01:36:59.434227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.434209 2579 scope.go:117] "RemoveContainer" containerID="e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" Sep 14 01:36:59.440836 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.440820 2579 scope.go:117] "RemoveContainer" containerID="590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" Sep 14 01:36:59.445089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.445069 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:36:59.448082 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.448051 2579 scope.go:117] "RemoveContainer" containerID="3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" Sep 14 01:36:59.448991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.448971 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:36:59.454304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.454286 2579 scope.go:117] "RemoveContainer" containerID="c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" Sep 14 01:36:59.460820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.460803 2579 scope.go:117] "RemoveContainer" containerID="45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" Sep 14 01:36:59.467858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.467844 2579 scope.go:117] "RemoveContainer" containerID="4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4" Sep 14 01:36:59.471526 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471506 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:36:59.471866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471848 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="prom-label-proxy" Sep 14 01:36:59.471866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471866 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="prom-label-proxy" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471882 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerName="registry" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471887 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerName="registry" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471898 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="config-reloader" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471904 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="config-reloader" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471909 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471913 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471921 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-web" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471925 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-web" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471932 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="alertmanager" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="alertmanager" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471943 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-metric" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471949 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-metric" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471954 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="init-config-reloader" Sep 14 01:36:59.471981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.471959 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="init-config-reloader" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472000 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-web" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472008 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="alertmanager" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472013 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472017 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="prom-label-proxy" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe257aad-79a3-4190-b3ea-7e79c576e610" containerName="registry" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="kube-rbac-proxy-metric" Sep 14 01:36:59.472602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.472034 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" containerName="config-reloader" Sep 14 01:36:59.476676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.476658 2579 scope.go:117] "RemoveContainer" containerID="35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" Sep 14 01:36:59.476944 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.476923 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5\": container with ID starting with 35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5 not found: ID does not exist" containerID="35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" Sep 14 01:36:59.476986 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.476952 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5"} err="failed to get container status \"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5\": rpc error: code = NotFound desc = could not find container \"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5\": container with ID starting with 35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5 not found: ID does not exist" Sep 14 01:36:59.476986 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.476969 2579 scope.go:117] "RemoveContainer" containerID="e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" Sep 14 01:36:59.477314 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.477298 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd\": container with ID starting with e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd not found: ID does not exist" containerID="e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" Sep 14 01:36:59.477376 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477318 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd"} err="failed to get container status \"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd\": rpc error: code = NotFound desc = could not find container \"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd\": container with ID starting with e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd not found: ID does not exist" Sep 14 01:36:59.477376 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477333 2579 scope.go:117] "RemoveContainer" containerID="590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" Sep 14 01:36:59.477542 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.477528 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0\": container with ID starting with 590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0 not found: ID does not exist" containerID="590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" Sep 14 01:36:59.477581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477545 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0"} err="failed to get container status \"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0\": rpc error: code = NotFound desc = could not find container \"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0\": container with ID starting with 590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0 not found: ID does not exist" Sep 14 01:36:59.477581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477557 2579 scope.go:117] "RemoveContainer" containerID="3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" Sep 14 01:36:59.477793 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.477776 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553\": container with ID starting with 3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553 not found: ID does not exist" containerID="3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" Sep 14 01:36:59.477863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477801 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553"} err="failed to get container status \"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553\": rpc error: code = NotFound desc = could not find container \"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553\": container with ID starting with 3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553 not found: ID does not exist" Sep 14 01:36:59.477863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.477821 2579 scope.go:117] "RemoveContainer" containerID="c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" Sep 14 01:36:59.478037 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.478019 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e\": container with ID starting with c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e not found: ID does not exist" containerID="c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" Sep 14 01:36:59.478108 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478040 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e"} err="failed to get container status \"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e\": rpc error: code = NotFound desc = could not find container \"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e\": container with ID starting with c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e not found: ID does not exist" Sep 14 01:36:59.478108 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478068 2579 scope.go:117] "RemoveContainer" containerID="45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" Sep 14 01:36:59.478275 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.478261 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7\": container with ID starting with 45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7 not found: ID does not exist" containerID="45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" Sep 14 01:36:59.478314 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478279 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7"} err="failed to get container status \"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7\": rpc error: code = NotFound desc = could not find container \"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7\": container with ID starting with 45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7 not found: ID does not exist" Sep 14 01:36:59.478314 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478292 2579 scope.go:117] "RemoveContainer" containerID="4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4" Sep 14 01:36:59.478395 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.478608 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:36:59.478581 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4\": container with ID starting with 4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4 not found: ID does not exist" containerID="4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4" Sep 14 01:36:59.478697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478611 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4"} err="failed to get container status \"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4\": rpc error: code = NotFound desc = could not find container \"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4\": container with ID starting with 4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4 not found: ID does not exist" Sep 14 01:36:59.478697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478633 2579 scope.go:117] "RemoveContainer" containerID="35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5" Sep 14 01:36:59.478964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478942 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5"} err="failed to get container status \"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5\": rpc error: code = NotFound desc = could not find container \"35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5\": container with ID starting with 35a35ca170fc0bb765bf8244e28dbe7927cfb577df95b00c359514efc58034a5 not found: ID does not exist" Sep 14 01:36:59.479022 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.478964 2579 scope.go:117] "RemoveContainer" containerID="e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd" Sep 14 01:36:59.479233 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479214 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd"} err="failed to get container status \"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd\": rpc error: code = NotFound desc = could not find container \"e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd\": container with ID starting with e0397d61ad56b6bbeece13fdbc156b4da35de128814106bee727ad542948bcdd not found: ID does not exist" Sep 14 01:36:59.479289 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479234 2579 scope.go:117] "RemoveContainer" containerID="590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0" Sep 14 01:36:59.479440 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479424 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0"} err="failed to get container status \"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0\": rpc error: code = NotFound desc = could not find container \"590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0\": container with ID starting with 590f370d4f0e6ac57bcade1003be753839f4a46049f3e100a1d70ca66d3d23f0 not found: ID does not exist" Sep 14 01:36:59.479481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479440 2579 scope.go:117] "RemoveContainer" containerID="3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553" Sep 14 01:36:59.479714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479662 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553"} err="failed to get container status \"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553\": rpc error: code = NotFound desc = could not find container \"3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553\": container with ID starting with 3a54c19689de5e1d7842923a7580dd99af5f60a0ec9b40fbbbb89a8b22fb9553 not found: ID does not exist" Sep 14 01:36:59.479714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479693 2579 scope.go:117] "RemoveContainer" containerID="c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e" Sep 14 01:36:59.479924 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479903 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e"} err="failed to get container status \"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e\": rpc error: code = NotFound desc = could not find container \"c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e\": container with ID starting with c2d1d9fde06c84cdf4099ba907f58f4d1a8d0c356cb96f9903dd612498284d0e not found: ID does not exist" Sep 14 01:36:59.479999 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.479926 2579 scope.go:117] "RemoveContainer" containerID="45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7" Sep 14 01:36:59.480220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480201 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7"} err="failed to get container status \"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7\": rpc error: code = NotFound desc = could not find container \"45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7\": container with ID starting with 45a85ed5b31771cc654ba16af05d0c5ad12f880e00fc137952531e19372d38d7 not found: ID does not exist" Sep 14 01:36:59.480276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480221 2579 scope.go:117] "RemoveContainer" containerID="4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4" Sep 14 01:36:59.480452 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480433 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4"} err="failed to get container status \"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4\": rpc error: code = NotFound desc = could not find container \"4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4\": container with ID starting with 4e50d12ac8df504c6430c4cdf83477c483f69e650cacb8ab684d62bf6c21eaf4 not found: ID does not exist" Sep 14 01:36:59.480587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480569 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 01:36:59.480627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480569 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 01:36:59.480849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.480832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 01:36:59.481104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481042 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 01:36:59.481170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 01:36:59.481238 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481222 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 01:36:59.481278 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 01:36:59.481378 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481362 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sxcm9\"" Sep 14 01:36:59.481479 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.481456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 01:36:59.485371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.485353 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 01:36:59.488589 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.488569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:36:59.605212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-web-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-config-out\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzw69\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-kube-api-access-dzw69\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.605752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.605631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.706991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.706914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.706991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.706959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.706991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.706982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-config-out\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzw69\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-kube-api-access-dzw69\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707424 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707477 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707454 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.707637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.707580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-web-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.708346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.708320 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.708686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.708660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.709346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.709319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b546adde-3637-4d5b-a529-902fe62ebed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.710326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.710956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b546adde-3637-4d5b-a529-902fe62ebed0-config-out\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.710956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.710956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.710956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.711237 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.710993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-web-config\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.711307 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.711290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.711377 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.711354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.712004 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.711984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b546adde-3637-4d5b-a529-902fe62ebed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.714842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.714824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzw69\" (UniqueName: \"kubernetes.io/projected/b546adde-3637-4d5b-a529-902fe62ebed0-kube-api-access-dzw69\") pod \"alertmanager-main-0\" (UID: \"b546adde-3637-4d5b-a529-902fe62ebed0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:36:59.788194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:36:59.788159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 01:37:00.066260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:00.066226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 01:37:00.069072 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:37:00.069037 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb546adde_3637_4d5b_a529_902fe62ebed0.slice/crio-1dd6d3ceb65a3f598eb72bf8bda1118ef78ed23fe783dcf22f38ef13aa29d0e7 WatchSource:0}: Error finding container 1dd6d3ceb65a3f598eb72bf8bda1118ef78ed23fe783dcf22f38ef13aa29d0e7: Status 404 returned error can't find the container with id 1dd6d3ceb65a3f598eb72bf8bda1118ef78ed23fe783dcf22f38ef13aa29d0e7 Sep 14 01:37:00.431419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:00.431338 2579 generic.go:356] "Generic (PLEG): container finished" podID="b546adde-3637-4d5b-a529-902fe62ebed0" containerID="fd6bf5d31c6438e50b3149226aec425c731d71462081974163435e5e99e66539" exitCode=0 Sep 14 01:37:00.431791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:00.431426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerDied","Data":"fd6bf5d31c6438e50b3149226aec425c731d71462081974163435e5e99e66539"} Sep 14 01:37:00.431791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:00.431501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"1dd6d3ceb65a3f598eb72bf8bda1118ef78ed23fe783dcf22f38ef13aa29d0e7"} Sep 14 01:37:01.209348 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.209313 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="495d8e9e-2c23-424f-a9bc-1b0714f3e8e2" path="/var/lib/kubelet/pods/495d8e9e-2c23-424f-a9bc-1b0714f3e8e2/volumes" Sep 14 01:37:01.439528 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"af588f9def274d915dd62823826b16cff361f6af573e3ef14d9edd4d937b7aee"} Sep 14 01:37:01.439971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"f637b23b1f87d88ccd033b892f5e93f6f2a7bad4cbf6d86149b6d5183b25df10"} Sep 14 01:37:01.439971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"77cf2f7de258c1b79beb74cd29db4b0f3d5a743877588921f946b8f92876a6eb"} Sep 14 01:37:01.439971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"63ff14e922c66fcf6b1483d11f77accc391a5a9c840b3f1fcb72c1ed30ef099b"} Sep 14 01:37:01.439971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"970372be64708708a31787678fcfccee8bd639a91d98278bc85effee1ef88924"} Sep 14 01:37:01.439971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.439673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b546adde-3637-4d5b-a529-902fe62ebed0","Type":"ContainerStarted","Data":"ae0c9734865f8c86c92f11c3a23a1ddbfc8e33395e5436cb9422e003ed952fed"} Sep 14 01:37:01.463556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:01.463465 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.448289627 podStartE2EDuration="2.463452612s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.162985ms" isStatefulPod=false podCreationTimestamp="2026-09-14 01:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:37:01.462917535 +0000 UTC m=+366.696282803" watchObservedRunningTime="2026-09-14 01:37:01.463452612 +0000 UTC m=+366.696817868" Sep 14 01:37:05.276553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:05.276519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:05.377811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:05.377773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:09.178366 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.178332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:37:09.180510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.180488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5450a90-678e-4192-ab47-81dcef253c6c-metrics-certs\") pod \"network-metrics-daemon-xhtx9\" (UID: \"d5450a90-678e-4192-ab47-81dcef253c6c\") " pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:37:09.306743 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.306718 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gv4zq\"" Sep 14 01:37:09.314864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.314840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xhtx9" Sep 14 01:37:09.510879 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.510847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:09.583774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.583745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xhtx9"] Sep 14 01:37:09.586709 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:37:09.586678 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5450a90_678e_4192_ab47_81dcef253c6c.slice/crio-4b6aea783a3b379fc49e83563b373571bb16f251a1e5d63ed489ff47e92830ee WatchSource:0}: Error finding container 4b6aea783a3b379fc49e83563b373571bb16f251a1e5d63ed489ff47e92830ee: Status 404 returned error can't find the container with id 4b6aea783a3b379fc49e83563b373571bb16f251a1e5d63ed489ff47e92830ee Sep 14 01:37:09.588655 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.588638 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:37:09.614023 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.613990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:09.669254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:09.669228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:10.357008 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:10.356905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:10.408322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:10.408282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:10.478387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:10.478362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xhtx9" event={"ID":"d5450a90-678e-4192-ab47-81dcef253c6c","Type":"ContainerStarted","Data":"4b6aea783a3b379fc49e83563b373571bb16f251a1e5d63ed489ff47e92830ee"} Sep 14 01:37:11.484004 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.483972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xhtx9" event={"ID":"d5450a90-678e-4192-ab47-81dcef253c6c","Type":"ContainerStarted","Data":"bf3ee2036630ff38586b76392ccf9d5eca0c19bc31895e5f4b1d2bd46c0500d8"} Sep 14 01:37:11.484390 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.484020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xhtx9" event={"ID":"d5450a90-678e-4192-ab47-81dcef253c6c","Type":"ContainerStarted","Data":"fb427e5afbb0cc70cba09f4bef6a9eb7e874f82a6a2d33ecc87d6a6b90912965"} Sep 14 01:37:11.499628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.499588 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xhtx9" podStartSLOduration=375.674770136 podStartE2EDuration="6m16.49957636s" totalImagesPullingTime="824.806224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:37:11.498401093 +0000 UTC m=+376.731766373" watchObservedRunningTime="2026-09-14 01:37:11.49957636 +0000 UTC m=+376.732941617" Sep 14 01:37:11.579115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.579084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:11.634703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.634666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:11.732990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:11.732955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:13.134718 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:13.134682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:13.235295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:13.235267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:44.166991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:44.166952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:44.213590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:44.213556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:51.803373 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:51.803333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:37:51.899846 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:37:51.899817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:13.354778 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:13.354738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:14.127110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:14.127072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:16.959744 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:16.959696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:17.007507 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:17.007456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:18.420993 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:18.420907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:18.520949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:18.520910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:19.106104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:19.106049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:19.209971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:19.209935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:20.419568 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:20.419535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:20.810840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:20.810807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:21.177232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.177158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk"] Sep 14 01:38:21.180431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.180413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.182632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.182612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 14 01:38:21.183036 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.183016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 14 01:38:21.183585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.183567 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4br7x\"" Sep 14 01:38:21.188728 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.188706 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk"] Sep 14 01:38:21.338412 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.338374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.338580 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.338425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqdh\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-kube-api-access-wfqdh\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.438975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.438879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqdh\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-kube-api-access-wfqdh\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.438975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.438971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.448146 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.448122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.448256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.448169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqdh\" (UniqueName: \"kubernetes.io/projected/9baddfd2-b22a-445a-8959-406b4c9a03fa-kube-api-access-wfqdh\") pod \"cert-manager-cainjector-559cdd8c87-2d2sk\" (UID: \"9baddfd2-b22a-445a-8959-406b4c9a03fa\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.490157 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.490123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" Sep 14 01:38:21.755471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:21.755442 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk"] Sep 14 01:38:21.757859 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:38:21.757818 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9baddfd2_b22a_445a_8959_406b4c9a03fa.slice/crio-7a0b566645f757cec956181ad246fcee9904458c4aaed8c7e8e6843fecc940ef WatchSource:0}: Error finding container 7a0b566645f757cec956181ad246fcee9904458c4aaed8c7e8e6843fecc940ef: Status 404 returned error can't find the container with id 7a0b566645f757cec956181ad246fcee9904458c4aaed8c7e8e6843fecc940ef Sep 14 01:38:22.746312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:22.746264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" event={"ID":"9baddfd2-b22a-445a-8959-406b4c9a03fa","Type":"ContainerStarted","Data":"7a0b566645f757cec956181ad246fcee9904458c4aaed8c7e8e6843fecc940ef"} Sep 14 01:38:23.763295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:23.763258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:23.817213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:23.817179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:26.763120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:26.763080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" event={"ID":"9baddfd2-b22a-445a-8959-406b4c9a03fa","Type":"ContainerStarted","Data":"a6c40d3f0e1a4bff1a784a4f6f05f95eff0a693f7b1e75cdd96088f06ded3a57"} Sep 14 01:38:26.778329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:26.778282 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2d2sk" podStartSLOduration=1.790141075 podStartE2EDuration="5.778268734s" totalImagesPullingTime="3.988127659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:38:26.776865967 +0000 UTC m=+452.010231247" watchObservedRunningTime="2026-09-14 01:38:26.778268734 +0000 UTC m=+452.011633990" Sep 14 01:38:29.219678 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:29.219649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:31.438861 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:31.438826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:31.538843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:31.538810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:33.159826 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.159787 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct"] Sep 14 01:38:33.163041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.163021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.165276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.165254 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fkbmt\"" Sep 14 01:38:33.165276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.165270 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:38:33.166171 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.166137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 01:38:33.167714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.167695 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct"] Sep 14 01:38:33.232803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.232774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fc0a50-cfb5-40ad-b513-8502721f2632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.232939 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.232807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qks\" (UniqueName: \"kubernetes.io/projected/63fc0a50-cfb5-40ad-b513-8502721f2632-kube-api-access-r6qks\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.333421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.333392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fc0a50-cfb5-40ad-b513-8502721f2632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.333544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.333428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qks\" (UniqueName: \"kubernetes.io/projected/63fc0a50-cfb5-40ad-b513-8502721f2632-kube-api-access-r6qks\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.333739 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.333719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fc0a50-cfb5-40ad-b513-8502721f2632-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.343730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.343704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qks\" (UniqueName: \"kubernetes.io/projected/63fc0a50-cfb5-40ad-b513-8502721f2632-kube-api-access-r6qks\") pod \"openshift-lws-operator-fd8ccff4c-fzfct\" (UID: \"63fc0a50-cfb5-40ad-b513-8502721f2632\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.473197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.473098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" Sep 14 01:38:33.748431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.748260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct"] Sep 14 01:38:33.751111 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:38:33.751084 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63fc0a50_cfb5_40ad_b513_8502721f2632.slice/crio-38a424d57ae61af5a51af64d87b9effe4e43d507fd51a077321a9fe55c19b02e WatchSource:0}: Error finding container 38a424d57ae61af5a51af64d87b9effe4e43d507fd51a077321a9fe55c19b02e: Status 404 returned error can't find the container with id 38a424d57ae61af5a51af64d87b9effe4e43d507fd51a077321a9fe55c19b02e Sep 14 01:38:33.790053 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:33.790021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" event={"ID":"63fc0a50-cfb5-40ad-b513-8502721f2632","Type":"ContainerStarted","Data":"38a424d57ae61af5a51af64d87b9effe4e43d507fd51a077321a9fe55c19b02e"} Sep 14 01:38:35.009304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:35.009273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:35.062285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:35.062240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:37.807519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:37.807480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" event={"ID":"63fc0a50-cfb5-40ad-b513-8502721f2632","Type":"ContainerStarted","Data":"453aeca8ec8f7654ec49fbcd654f0e74bd242a74d6dca1e413031d75fb0488d6"} Sep 14 01:38:37.822733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:37.822682 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fzfct" podStartSLOduration=1.147427877 podStartE2EDuration="4.822664685s" totalImagesPullingTime="3.675236808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:38:37.821457258 +0000 UTC m=+463.054822530" watchObservedRunningTime="2026-09-14 01:38:37.822664685 +0000 UTC m=+463.056029942" Sep 14 01:38:38.500027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:38.499974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:38.600903 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:38.600866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:40.667212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:40.667175 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:56.597211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:56.597173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:56.695101 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:56.695046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.197642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.197585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.395614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.395578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.493774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.493727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.548019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.547985 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl"] Sep 14 01:38:57.553605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.553580 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.557032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.556908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 01:38:57.557497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.557473 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kpfz9\"" Sep 14 01:38:57.557945 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.557929 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 01:38:57.558267 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.558253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 01:38:57.558575 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.558558 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 01:38:57.568163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.568139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl"] Sep 14 01:38:57.598997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.598960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.599457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.599005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.599457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.599081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.599457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.599107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.599457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.599196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsmt\" (UniqueName: \"kubernetes.io/projected/ec030c39-7709-474e-b00d-894aede051f0-kube-api-access-swsmt\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.698636 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.698600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.699635 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.699610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.699774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.699642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.699774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.699678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.699774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.699693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.699774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.699716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsmt\" (UniqueName: \"kubernetes.io/projected/ec030c39-7709-474e-b00d-894aede051f0-kube-api-access-swsmt\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.700144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.700123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.702119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.702095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec030c39-7709-474e-b00d-894aede051f0-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.702226 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.702201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.702285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.702260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec030c39-7709-474e-b00d-894aede051f0-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.707272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.707250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsmt\" (UniqueName: \"kubernetes.io/projected/ec030c39-7709-474e-b00d-894aede051f0-kube-api-access-swsmt\") pod \"opendatahub-operator-controller-manager-c5d46b944-mxcgl\" (UID: \"ec030c39-7709-474e-b00d-894aede051f0\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.801806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.801720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.866817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.866782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:38:57.898487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.898437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:57.999653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:57.999618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:58.099401 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:58.099308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:58.145148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:58.145112 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl"] Sep 14 01:38:58.148822 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:38:58.148797 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec030c39_7709_474e_b00d_894aede051f0.slice/crio-0509a2b3de76473255bb6e8aa64a828497390d78f1b3274fe287c0b7f9d759d5 WatchSource:0}: Error finding container 0509a2b3de76473255bb6e8aa64a828497390d78f1b3274fe287c0b7f9d759d5: Status 404 returned error can't find the container with id 0509a2b3de76473255bb6e8aa64a828497390d78f1b3274fe287c0b7f9d759d5 Sep 14 01:38:58.199609 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:58.199571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:38:58.893260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:38:58.893220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" event={"ID":"ec030c39-7709-474e-b00d-894aede051f0","Type":"ContainerStarted","Data":"0509a2b3de76473255bb6e8aa64a828497390d78f1b3274fe287c0b7f9d759d5"} Sep 14 01:39:01.909457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:01.909415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" event={"ID":"ec030c39-7709-474e-b00d-894aede051f0","Type":"ContainerStarted","Data":"813bbac470cb800f6c9b9c4d69791f1bc643778ac6a4e321ebe999316bb95aa0"} Sep 14 01:39:02.888751 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:02.888702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:02.915641 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:02.915611 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec030c39-7709-474e-b00d-894aede051f0" containerID="813bbac470cb800f6c9b9c4d69791f1bc643778ac6a4e321ebe999316bb95aa0" exitCode=0 Sep 14 01:39:02.916112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:02.915706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" event={"ID":"ec030c39-7709-474e-b00d-894aede051f0","Type":"ContainerDied","Data":"813bbac470cb800f6c9b9c4d69791f1bc643778ac6a4e321ebe999316bb95aa0"} Sep 14 01:39:02.944963 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:02.944915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:03.922135 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:03.922097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" event={"ID":"ec030c39-7709-474e-b00d-894aede051f0","Type":"ContainerStarted","Data":"a5c0e75f2bf627621e666ce5d6ce65b18e31013daf5f9100f9636a3c488bad0a"} Sep 14 01:39:03.922610 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:03.922212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:39:03.944993 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:03.944944 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" podStartSLOduration=2.487706716 podStartE2EDuration="6.944928782s" totalImagesPullingTime="3.404059931s" totalInitContainerRuntime="1.053162135s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:38:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:03.942681655 +0000 UTC m=+489.176046932" watchObservedRunningTime="2026-09-14 01:39:03.944928782 +0000 UTC m=+489.178294039" Sep 14 01:39:07.010932 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:07.010891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:07.070981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:07.070942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:09.656994 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:09.656958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:09.756526 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:09.756485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:14.928624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:14.928546 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-mxcgl" Sep 14 01:39:16.521312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.521268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:16.623305 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.623261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:16.798790 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.798689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:16.965667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.965637 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg"] Sep 14 01:39:16.974226 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.974203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:16.977736 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.977711 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 01:39:16.977878 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.977773 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 14 01:39:16.978570 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.978547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b7s7l\"" Sep 14 01:39:16.979011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.978827 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 14 01:39:16.980118 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.980098 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 01:39:16.980818 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:16.980797 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg"] Sep 14 01:39:17.061453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.061370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9426f359-66ae-482d-8573-bad9ffb4cb48-tmp\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.061453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.061407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9426f359-66ae-482d-8573-bad9ffb4cb48-tls-certs\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.061453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.061428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chj5c\" (UniqueName: \"kubernetes.io/projected/9426f359-66ae-482d-8573-bad9ffb4cb48-kube-api-access-chj5c\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.162459 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.162421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9426f359-66ae-482d-8573-bad9ffb4cb48-tmp\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.162459 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.162458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9426f359-66ae-482d-8573-bad9ffb4cb48-tls-certs\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.162671 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.162480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chj5c\" (UniqueName: \"kubernetes.io/projected/9426f359-66ae-482d-8573-bad9ffb4cb48-kube-api-access-chj5c\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.164737 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.164708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9426f359-66ae-482d-8573-bad9ffb4cb48-tmp\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.164931 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.164914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9426f359-66ae-482d-8573-bad9ffb4cb48-tls-certs\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.168236 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.168208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:17.170746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.170725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chj5c\" (UniqueName: \"kubernetes.io/projected/9426f359-66ae-482d-8573-bad9ffb4cb48-kube-api-access-chj5c\") pod \"kube-auth-proxy-7cb7b7df59-cm6xg\" (UID: \"9426f359-66ae-482d-8573-bad9ffb4cb48\") " pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.284541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.284509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" Sep 14 01:39:17.560043 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.557551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg"] Sep 14 01:39:17.563049 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:39:17.563012 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9426f359_66ae_482d_8573_bad9ffb4cb48.slice/crio-5fefcc8ed8465dabcd3c739780a805cabd529cfbdb68252537f1ae4f1c39c506 WatchSource:0}: Error finding container 5fefcc8ed8465dabcd3c739780a805cabd529cfbdb68252537f1ae4f1c39c506: Status 404 returned error can't find the container with id 5fefcc8ed8465dabcd3c739780a805cabd529cfbdb68252537f1ae4f1c39c506 Sep 14 01:39:17.988321 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:17.988286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" event={"ID":"9426f359-66ae-482d-8573-bad9ffb4cb48","Type":"ContainerStarted","Data":"5fefcc8ed8465dabcd3c739780a805cabd529cfbdb68252537f1ae4f1c39c506"} Sep 14 01:39:18.211655 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:18.211595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:18.274740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:18.274568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:19.135916 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:19.135876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:21.003182 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:21.003151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" event={"ID":"9426f359-66ae-482d-8573-bad9ffb4cb48","Type":"ContainerStarted","Data":"7f7a23b643f0fa8723272e835ad91edbdf934cbfae9791366e6b8fade5b95925"} Sep 14 01:39:21.018333 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:21.018282 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cb7b7df59-cm6xg" podStartSLOduration=2.2707640270000002 podStartE2EDuration="5.018268411s" totalImagesPullingTime="2.747504384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:21.016853025 +0000 UTC m=+506.250218281" watchObservedRunningTime="2026-09-14 01:39:21.018268411 +0000 UTC m=+506.251633667" Sep 14 01:39:24.561550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:24.561519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:25.276615 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.276578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:25.400247 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.400208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:25.927587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.927550 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n"] Sep 14 01:39:25.930783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.930764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:25.933115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.933095 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 01:39:25.933203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.933104 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wxdbs\"" Sep 14 01:39:25.942919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.942890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n"] Sep 14 01:39:25.958267 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:25.958242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:26.040177 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040177 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78nd\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-kube-api-access-t78nd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf63e43-a015-4240-a64b-66885f55222d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf63e43-a015-4240-a64b-66885f55222d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.040541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.040413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.095521 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.095484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:26.141376 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78nd\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-kube-api-access-t78nd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf63e43-a015-4240-a64b-66885f55222d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf63e43-a015-4240-a64b-66885f55222d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.141970 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141943 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.142021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.141741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.142095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.142029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.142160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.142128 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.142241 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.142223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf63e43-a015-4240-a64b-66885f55222d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.143682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.143661 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf63e43-a015-4240-a64b-66885f55222d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.144007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.143990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf63e43-a015-4240-a64b-66885f55222d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.150010 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.149989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.150729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.150711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78nd\" (UniqueName: \"kubernetes.io/projected/abf63e43-a015-4240-a64b-66885f55222d-kube-api-access-t78nd\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n\" (UID: \"abf63e43-a015-4240-a64b-66885f55222d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.196437 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.196376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:26.241546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.241520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:26.512714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.512692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n"] Sep 14 01:39:26.514637 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:39:26.514605 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf63e43_a015_4240_a64b_66885f55222d.slice/crio-c819990344938c50fde354a6cb280475dda11ef87a6974aa298d0fe6ca348366 WatchSource:0}: Error finding container c819990344938c50fde354a6cb280475dda11ef87a6974aa298d0fe6ca348366: Status 404 returned error can't find the container with id c819990344938c50fde354a6cb280475dda11ef87a6974aa298d0fe6ca348366 Sep 14 01:39:26.566809 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.566774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:26.615810 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:26.615778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:27.028909 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.028877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" event={"ID":"abf63e43-a015-4240-a64b-66885f55222d","Type":"ContainerStarted","Data":"c819990344938c50fde354a6cb280475dda11ef87a6974aa298d0fe6ca348366"} Sep 14 01:39:27.538034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.538004 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcgrr"] Sep 14 01:39:27.541916 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.541790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.549081 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.548901 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 14 01:39:27.549258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.549172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mfs7j\"" Sep 14 01:39:27.564031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.564005 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcgrr"] Sep 14 01:39:27.578606 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.578566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:27.655791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.655751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49h4j\" (UniqueName: \"kubernetes.io/projected/c63a1f82-cca1-4079-b45b-b108955c1fb3-kube-api-access-49h4j\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.655975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.655826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.662947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.662916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:27.757224 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.757185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49h4j\" (UniqueName: \"kubernetes.io/projected/c63a1f82-cca1-4079-b45b-b108955c1fb3-kube-api-access-49h4j\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.757374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.757341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.757510 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:39:27.757488 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 14 01:39:27.757615 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:39:27.757561 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert podName:c63a1f82-cca1-4079-b45b-b108955c1fb3 nodeName:}" failed. No retries permitted until 2026-09-14 01:39:28.257538185 +0000 UTC m=+513.490903485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert") pod "odh-model-controller-7998c67dc4-zcgrr" (UID: "c63a1f82-cca1-4079-b45b-b108955c1fb3") : secret "odh-model-controller-webhook-cert" not found Sep 14 01:39:27.767761 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.767736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49h4j\" (UniqueName: \"kubernetes.io/projected/c63a1f82-cca1-4079-b45b-b108955c1fb3-kube-api-access-49h4j\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:27.908251 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.908157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:27.930172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.930144 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5fngv"] Sep 14 01:39:27.933349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.933334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:27.935581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.935549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 14 01:39:27.935703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.935606 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5np2g\"" Sep 14 01:39:27.941589 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:27.941434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5fngv"] Sep 14 01:39:28.024281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.024250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:28.060186 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.060151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbd6\" (UniqueName: \"kubernetes.io/projected/3328ae26-849a-41b2-9b10-a7553c33225e-kube-api-access-5qbd6\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.060573 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.060334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3328ae26-849a-41b2-9b10-a7553c33225e-tls-certs\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.083151 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.083011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:28.161574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.161495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3328ae26-849a-41b2-9b10-a7553c33225e-tls-certs\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.161574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.161549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbd6\" (UniqueName: \"kubernetes.io/projected/3328ae26-849a-41b2-9b10-a7553c33225e-kube-api-access-5qbd6\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.164599 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.164569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3328ae26-849a-41b2-9b10-a7553c33225e-tls-certs\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.169473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.169438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbd6\" (UniqueName: \"kubernetes.io/projected/3328ae26-849a-41b2-9b10-a7553c33225e-kube-api-access-5qbd6\") pod \"model-serving-api-54d89bd5fc-5fngv\" (UID: \"3328ae26-849a-41b2-9b10-a7553c33225e\") " pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.246496 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.246468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:28.262231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.262204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:28.264638 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.264613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c63a1f82-cca1-4079-b45b-b108955c1fb3-cert\") pod \"odh-model-controller-7998c67dc4-zcgrr\" (UID: \"c63a1f82-cca1-4079-b45b-b108955c1fb3\") " pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:28.462201 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.462127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:28.515271 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.515242 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5fngv"] Sep 14 01:39:28.519604 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:39:28.519566 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3328ae26_849a_41b2_9b10_a7553c33225e.slice/crio-db49190a195de7d32ce907d4f5af88238963b25834a8c2ea37ecc3c60deed5bb WatchSource:0}: Error finding container db49190a195de7d32ce907d4f5af88238963b25834a8c2ea37ecc3c60deed5bb: Status 404 returned error can't find the container with id db49190a195de7d32ce907d4f5af88238963b25834a8c2ea37ecc3c60deed5bb Sep 14 01:39:28.740428 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:28.740332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zcgrr"] Sep 14 01:39:29.040108 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:29.040005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" event={"ID":"3328ae26-849a-41b2-9b10-a7553c33225e","Type":"ContainerStarted","Data":"db49190a195de7d32ce907d4f5af88238963b25834a8c2ea37ecc3c60deed5bb"} Sep 14 01:39:29.695543 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:39:29.695511 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63a1f82_cca1_4079_b45b_b108955c1fb3.slice/crio-99e3a4c02399bd6bb241259411f182116d019014b60928c7077d6731c415ef3a WatchSource:0}: Error finding container 99e3a4c02399bd6bb241259411f182116d019014b60928c7077d6731c415ef3a: Status 404 returned error can't find the container with id 99e3a4c02399bd6bb241259411f182116d019014b60928c7077d6731c415ef3a Sep 14 01:39:29.755544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:29.755509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:39:29.755635 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:29.755598 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:39:29.755705 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:29.755641 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:39:30.049574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.049535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" event={"ID":"abf63e43-a015-4240-a64b-66885f55222d","Type":"ContainerStarted","Data":"c36fa7f821e061acfe4bb1907e80fa034b95fc3693f1416d4875a5a8248d2a0b"} Sep 14 01:39:30.051128 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.051096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" event={"ID":"c63a1f82-cca1-4079-b45b-b108955c1fb3","Type":"ContainerStarted","Data":"99e3a4c02399bd6bb241259411f182116d019014b60928c7077d6731c415ef3a"} Sep 14 01:39:30.068886 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.068837 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" podStartSLOduration=1.830619003 podStartE2EDuration="5.068817753s" totalImagesPullingTime="3.23819875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:30.0671037 +0000 UTC m=+515.300468981" watchObservedRunningTime="2026-09-14 01:39:30.068817753 +0000 UTC m=+515.302183011" Sep 14 01:39:30.242480 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.242442 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:30.244320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.244291 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Sep 14 01:39:30.244441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:30.244352 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" podUID="abf63e43-a015-4240-a64b-66885f55222d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Sep 14 01:39:31.059938 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:31.059898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" event={"ID":"3328ae26-849a-41b2-9b10-a7553c33225e","Type":"ContainerStarted","Data":"8a89f987d9ac6ba552f51bb9c72f77e72e8f23958923ef8ff501d164eaa5d141"} Sep 14 01:39:31.060405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:31.060143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:31.076528 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:31.076487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" podStartSLOduration=1.686139624 podStartE2EDuration="4.076467856s" totalImagesPullingTime="2.390328232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:31.074746406 +0000 UTC m=+516.308111698" watchObservedRunningTime="2026-09-14 01:39:31.076467856 +0000 UTC m=+516.309833166" Sep 14 01:39:31.246035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:31.246004 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:32.066370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:32.066340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:32.067517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:32.067493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n" Sep 14 01:39:32.615222 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:32.615187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:32.670299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:32.670254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:33.072499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:33.072462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" event={"ID":"c63a1f82-cca1-4079-b45b-b108955c1fb3","Type":"ContainerStarted","Data":"acd5a6cd46e1ba9ae811e4598bdeada0ef87be3ac4d4c68dd28130eb3bfe0646"} Sep 14 01:39:33.098910 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:33.098863 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" podStartSLOduration=2.948668664 podStartE2EDuration="6.098846471s" totalImagesPullingTime="3.150177807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:33.096906437 +0000 UTC m=+518.330271717" watchObservedRunningTime="2026-09-14 01:39:33.098846471 +0000 UTC m=+518.332211729" Sep 14 01:39:34.077523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.077483 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:34.170588 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.170552 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl"] Sep 14 01:39:34.176753 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.176734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.179457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.179435 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 14 01:39:34.179561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.179455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 14 01:39:34.179561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.179455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wbbvm\"" Sep 14 01:39:34.179664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.179587 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 14 01:39:34.187016 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.186997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl"] Sep 14 01:39:34.322606 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvmq\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-kube-api-access-cjvmq\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb8325c-5c39-46bc-bab7-94190f37ef43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.322918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.322832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb8325c-5c39-46bc-bab7-94190f37ef43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.423930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.423725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvmq\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-kube-api-access-cjvmq\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.424492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.424466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.426508 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.426477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb8325c-5c39-46bc-bab7-94190f37ef43-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.426650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.426630 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.426650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.426641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.426732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.426651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aeb8325c-5c39-46bc-bab7-94190f37ef43-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.437896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.437875 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvmq\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-kube-api-access-cjvmq\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.441764 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.441730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb8325c-5c39-46bc-bab7-94190f37ef43-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hx8sl\" (UID: \"aeb8325c-5c39-46bc-bab7-94190f37ef43\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.486528 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.486494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:34.765587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:34.765563 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl"] Sep 14 01:39:34.767907 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:39:34.767880 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb8325c_5c39_46bc_bab7_94190f37ef43.slice/crio-7fc82528e1bb6edae098c2cecd7de6150a30e7a6ea004749b831d5fd1ab5bb0c WatchSource:0}: Error finding container 7fc82528e1bb6edae098c2cecd7de6150a30e7a6ea004749b831d5fd1ab5bb0c: Status 404 returned error can't find the container with id 7fc82528e1bb6edae098c2cecd7de6150a30e7a6ea004749b831d5fd1ab5bb0c Sep 14 01:39:35.082518 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:35.082441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" event={"ID":"aeb8325c-5c39-46bc-bab7-94190f37ef43","Type":"ContainerStarted","Data":"7fc82528e1bb6edae098c2cecd7de6150a30e7a6ea004749b831d5fd1ab5bb0c"} Sep 14 01:39:36.572117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:36.572035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:36.660561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:36.660511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:36.949640 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:36.949557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:37.080566 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:37.080521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:37.340720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:37.340678 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:39:37.340811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:37.340751 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:39:37.889883 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:37.889839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:37.984724 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:37.984689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:38.046781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.046749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:38.097755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.097726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" event={"ID":"aeb8325c-5c39-46bc-bab7-94190f37ef43","Type":"ContainerStarted","Data":"d099c71d6c6b03aafac4d6fdd34c498acac232033e7c4542ca1690a311e2c045"} Sep 14 01:39:38.097985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.097968 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:38.099459 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.099433 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-hx8sl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:39:38.099557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.099485 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" podUID="aeb8325c-5c39-46bc-bab7-94190f37ef43" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:39:38.119487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:38.119448 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" podStartSLOduration=1.549097378 podStartE2EDuration="4.119432935s" totalImagesPullingTime="2.570335557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:38.118200047 +0000 UTC m=+523.351565304" watchObservedRunningTime="2026-09-14 01:39:38.119432935 +0000 UTC m=+523.352798192" Sep 14 01:39:39.103450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:39.103419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hx8sl" Sep 14 01:39:42.069899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:42.069867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5fngv" Sep 14 01:39:43.133698 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:43.133659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:43.234742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:43.234705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:39:45.105391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:45.105363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zcgrr" Sep 14 01:39:59.390186 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:39:59.390098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:12.657645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:12.657604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:28.938728 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:28.938691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:28.995103 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:28.995045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:37.514857 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:37.514817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:37.616700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:37.616653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:41.241163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:41.241119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:42.059273 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:42.059238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:42.429319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:42.429234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:42.519944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:42.519907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:42.845276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:42.845243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:45.709416 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.709379 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr"] Sep 14 01:40:45.711662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.711642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.714259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.714234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 01:40:45.714379 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.714280 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 01:40:45.714379 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.714234 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dccbm\"" Sep 14 01:40:45.726632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.726610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr"] Sep 14 01:40:45.837677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.837644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.837850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.837699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62qn\" (UniqueName: \"kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.938447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.938403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.938632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.938465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62qn\" (UniqueName: \"kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.938820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.938798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:45.948318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:45.948287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62qn\" (UniqueName: \"kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:46.021607 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:46.021576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:46.121566 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:46.121539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:46.304122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:46.304022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr"] Sep 14 01:40:46.307415 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:40:46.307388 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb005e2_3622_44b3_8c71_cb105436ceae.slice/crio-a44d50c280a7353e988aee397ed225a43ad0c1957ffd8b2fa0fa0ffd138f2847 WatchSource:0}: Error finding container a44d50c280a7353e988aee397ed225a43ad0c1957ffd8b2fa0fa0ffd138f2847: Status 404 returned error can't find the container with id a44d50c280a7353e988aee397ed225a43ad0c1957ffd8b2fa0fa0ffd138f2847 Sep 14 01:40:46.391537 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:46.391501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" event={"ID":"9eb005e2-3622-44b3-8c71-cb105436ceae","Type":"ContainerStarted","Data":"a44d50c280a7353e988aee397ed225a43ad0c1957ffd8b2fa0fa0ffd138f2847"} Sep 14 01:40:47.497207 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:47.497142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:48.258842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:48.258809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:48.536298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:48.536220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:49.906006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:49.905963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:50.057254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:50.057206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:50.661258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:50.661210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:51.418460 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:51.418423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" event={"ID":"9eb005e2-3622-44b3-8c71-cb105436ceae","Type":"ContainerStarted","Data":"755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9"} Sep 14 01:40:51.418841 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:51.418560 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:40:51.438120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:51.438053 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" podStartSLOduration=1.469780397 podStartE2EDuration="6.438039665s" totalImagesPullingTime="4.968259268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:40:51.436364404 +0000 UTC m=+596.669729684" watchObservedRunningTime="2026-09-14 01:40:51.438039665 +0000 UTC m=+596.671404923" Sep 14 01:40:52.105596 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.105559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:52.555611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.555575 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sbs6s"] Sep 14 01:40:52.558018 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.557999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:52.560292 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.560272 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tlgk5\"" Sep 14 01:40:52.570199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.570177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sbs6s"] Sep 14 01:40:52.706514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.706470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsdq\" (UniqueName: \"kubernetes.io/projected/f6e1e456-1af0-4374-b430-43b2088e9486-kube-api-access-8hsdq\") pod \"authorino-operator-76b478bd78-sbs6s\" (UID: \"f6e1e456-1af0-4374-b430-43b2088e9486\") " pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:52.807356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.807248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsdq\" (UniqueName: \"kubernetes.io/projected/f6e1e456-1af0-4374-b430-43b2088e9486-kube-api-access-8hsdq\") pod \"authorino-operator-76b478bd78-sbs6s\" (UID: \"f6e1e456-1af0-4374-b430-43b2088e9486\") " pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:52.819616 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.819583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsdq\" (UniqueName: \"kubernetes.io/projected/f6e1e456-1af0-4374-b430-43b2088e9486-kube-api-access-8hsdq\") pod \"authorino-operator-76b478bd78-sbs6s\" (UID: \"f6e1e456-1af0-4374-b430-43b2088e9486\") " pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:52.867094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:52.867050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:53.140447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.140423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sbs6s"] Sep 14 01:40:53.142557 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:40:53.142524 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e1e456_1af0_4374_b430_43b2088e9486.slice/crio-3d154336abfa037410a336fd125e9ec939e3f7c24275a71c5537908694a4950c WatchSource:0}: Error finding container 3d154336abfa037410a336fd125e9ec939e3f7c24275a71c5537908694a4950c: Status 404 returned error can't find the container with id 3d154336abfa037410a336fd125e9ec939e3f7c24275a71c5537908694a4950c Sep 14 01:40:53.428630 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.428551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" event={"ID":"f6e1e456-1af0-4374-b430-43b2088e9486","Type":"ContainerStarted","Data":"3d154336abfa037410a336fd125e9ec939e3f7c24275a71c5537908694a4950c"} Sep 14 01:40:53.696257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.696169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:53.796459 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.796408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:53.899959 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.899927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:53.981393 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:53.981359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:54.333562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:54.333495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:55.166374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:55.166348 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:40:55.167183 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:55.167165 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:40:56.444121 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.444080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" event={"ID":"f6e1e456-1af0-4374-b430-43b2088e9486","Type":"ContainerStarted","Data":"8bb996ff4fa57e16212e110f5a3b8426d47041b618f0686f33b0dc0a70dfe5a0"} Sep 14 01:40:56.444586 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.444144 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:40:56.460844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.460798 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" podStartSLOduration=1.768812049 podStartE2EDuration="4.460782827s" totalImagesPullingTime="2.691970778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:40:56.459082183 +0000 UTC m=+601.692447457" watchObservedRunningTime="2026-09-14 01:40:56.460782827 +0000 UTC m=+601.694148083" Sep 14 01:40:56.468148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.468123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:56.565304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.565270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:56.863947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.863916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:56.965262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:56.965215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:57.254682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:57.254644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:40:57.356474 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:40:57.356442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:02.425216 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:02.425180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:02.977735 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:02.977689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:03.080175 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:03.080141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:03.958438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:03.958401 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk"] Sep 14 01:41:03.962474 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:03.962446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:03.974847 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:03.974821 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk"] Sep 14 01:41:04.110225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.110189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.110390 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.110312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpldx\" (UniqueName: \"kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.169537 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.169506 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk"] Sep 14 01:41:04.169782 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.169763 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-mpldx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" Sep 14 01:41:04.201133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.201101 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686"] Sep 14 01:41:04.203640 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.203620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.210867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.210845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.210962 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.210947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpldx\" (UniqueName: \"kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.211269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.211250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.220409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.220389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686"] Sep 14 01:41:04.224622 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.224597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpldx\" (UniqueName: \"kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx\") pod \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.312197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.312171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl76g\" (UniqueName: \"kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.312313 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.312208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.413564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.413527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl76g\" (UniqueName: \"kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.413564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.413571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.413990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.413971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.421653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.421633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl76g\" (UniqueName: \"kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.463223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.463158 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk"] Sep 14 01:41:04.470167 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.470142 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk"] Sep 14 01:41:04.478357 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.478334 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr"] Sep 14 01:41:04.478554 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.478514 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" containerName="manager" containerID="cri-o://755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9" gracePeriod=2 Sep 14 01:41:04.480696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.480677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.488120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.488052 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr"] Sep 14 01:41:04.489298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.489277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.494923 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.494742 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9eb005e2-3622-44b3-8c71-cb105436ceae\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.498083 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.497942 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.500825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.500710 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686"] Sep 14 01:41:04.501041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.501021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:04.502205 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.502176 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.503133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.502893 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:04.503576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.503555 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" containerName="manager" Sep 14 01:41:04.503648 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.503580 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" containerName="manager" Sep 14 01:41:04.503698 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.503678 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" containerName="manager" Sep 14 01:41:04.505682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.505662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.506513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.506492 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686"] Sep 14 01:41:04.519541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.519153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:04.524870 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.524840 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:04.527513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.527491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.545384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.544586 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:04.547843 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.547810 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.550434 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.550407 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.586847 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.586806 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.589193 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.589160 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.614527 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614500 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpldx\" (UniqueName: \"kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx\") pod \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " Sep 14 01:41:04.614690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume\") pod \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\" (UID: \"2e3e40e4-28a2-4f15-9858-6d0b0a6345ee\") " Sep 14 01:41:04.614801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614773 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" (UID: "2e3e40e4-28a2-4f15-9858-6d0b0a6345ee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:41:04.614862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82h8m\" (UniqueName: \"kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.614911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.614960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.614931 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:04.617256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.616680 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx" (OuterVolumeSpecName: "kube-api-access-mpldx") pod "2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" (UID: "2e3e40e4-28a2-4f15-9858-6d0b0a6345ee"). InnerVolumeSpecName "kube-api-access-mpldx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:04.696306 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.696283 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.701050 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.701024 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:04.703585 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:04.703561 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:04.715594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.715536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.715681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.715659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82h8m\" (UniqueName: \"kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.715736 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.715691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4sqv\" (UniqueName: \"kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.715736 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.715729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.715831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.715762 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpldx\" (UniqueName: \"kubernetes.io/projected/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee-kube-api-access-mpldx\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:04.716052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.716031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.728527 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.728506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82h8m\" (UniqueName: \"kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dsbxv\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.816649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.816615 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume\") pod \"9eb005e2-3622-44b3-8c71-cb105436ceae\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " Sep 14 01:41:04.816805 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.816657 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b62qn\" (UniqueName: \"kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn\") pod \"9eb005e2-3622-44b3-8c71-cb105436ceae\" (UID: \"9eb005e2-3622-44b3-8c71-cb105436ceae\") " Sep 14 01:41:04.816871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.816845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.816990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.816971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4sqv\" (UniqueName: \"kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.817168 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.817141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9eb005e2-3622-44b3-8c71-cb105436ceae" (UID: "9eb005e2-3622-44b3-8c71-cb105436ceae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:41:04.817291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.817271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.818846 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.818826 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn" (OuterVolumeSpecName: "kube-api-access-b62qn") pod "9eb005e2-3622-44b3-8c71-cb105436ceae" (UID: "9eb005e2-3622-44b3-8c71-cb105436ceae"). InnerVolumeSpecName "kube-api-access-b62qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:04.829328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.829311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4sqv\" (UniqueName: \"kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ln4mm\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.859181 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.859159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:04.865129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.865109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:04.917888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.917842 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9eb005e2-3622-44b3-8c71-cb105436ceae-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:04.918003 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:04.917893 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b62qn\" (UniqueName: \"kubernetes.io/projected/9eb005e2-3622-44b3-8c71-cb105436ceae-kube-api-access-b62qn\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:05.130447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.130421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:05.133499 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:41:05.133474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd747025a_ee60_4587_bba1_fa886b67db53.slice/crio-0afb3eed82963401719aa9dc78606bc87651e4c818393eaa907386360b4ffa0b WatchSource:0}: Error finding container 0afb3eed82963401719aa9dc78606bc87651e4c818393eaa907386360b4ffa0b: Status 404 returned error can't find the container with id 0afb3eed82963401719aa9dc78606bc87651e4c818393eaa907386360b4ffa0b Sep 14 01:41:05.136896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.136876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:05.138774 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:41:05.138749 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960281cf_f6ee_434e_836b_84d261f93766.slice/crio-e1dac0ff1990266e910f0f79f28a2f9c10553311ec8eecad65d1204817ef1f1e WatchSource:0}: Error finding container e1dac0ff1990266e910f0f79f28a2f9c10553311ec8eecad65d1204817ef1f1e: Status 404 returned error can't find the container with id e1dac0ff1990266e910f0f79f28a2f9c10553311ec8eecad65d1204817ef1f1e Sep 14 01:41:05.207112 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:05.207075 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9pwxr\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:05.208925 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:05.208894 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:05.211187 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.211158 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" path="/var/lib/kubelet/pods/2e3e40e4-28a2-4f15-9858-6d0b0a6345ee/volumes" Sep 14 01:41:05.211571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.211551 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb005e2-3622-44b3-8c71-cb105436ceae" path="/var/lib/kubelet/pods/9eb005e2-3622-44b3-8c71-cb105436ceae/volumes" Sep 14 01:41:05.486812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.486782 2579 generic.go:356] "Generic (PLEG): container finished" podID="9eb005e2-3622-44b3-8c71-cb105436ceae" containerID="755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9" exitCode=0 Sep 14 01:41:05.486996 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.486834 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9pwxr" Sep 14 01:41:05.486996 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.486889 2579 scope.go:117] "RemoveContainer" containerID="755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9" Sep 14 01:41:05.488866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.488820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" event={"ID":"d747025a-ee60-4587-bba1-fa886b67db53","Type":"ContainerStarted","Data":"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6"} Sep 14 01:41:05.489097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.488918 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:05.489207 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.489125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" event={"ID":"d747025a-ee60-4587-bba1-fa886b67db53","Type":"ContainerStarted","Data":"0afb3eed82963401719aa9dc78606bc87651e4c818393eaa907386360b4ffa0b"} Sep 14 01:41:05.490808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.490785 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" event={"ID":"960281cf-f6ee-434e-836b-84d261f93766","Type":"ContainerStarted","Data":"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b"} Sep 14 01:41:05.490908 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.490830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:05.490908 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.490844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:05.490908 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.490854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" event={"ID":"960281cf-f6ee-434e-836b-84d261f93766","Type":"ContainerStarted","Data":"e1dac0ff1990266e910f0f79f28a2f9c10553311ec8eecad65d1204817ef1f1e"} Sep 14 01:41:05.496631 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.496615 2579 scope.go:117] "RemoveContainer" containerID="755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9" Sep 14 01:41:05.496889 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:05.496871 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9\": container with ID starting with 755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9 not found: ID does not exist" containerID="755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9" Sep 14 01:41:05.496944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.496902 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9"} err="failed to get container status \"755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9\": rpc error: code = NotFound desc = could not find container \"755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9\": container with ID starting with 755f32f3ada3c9df97f4a7921fa5fd43cfc5a1e7df9d5ff148fa740e5322e5d9 not found: ID does not exist" Sep 14 01:41:05.510946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.510883 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" podStartSLOduration=1.51086079 podStartE2EDuration="1.51086079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:05.506947854 +0000 UTC m=+610.740313193" watchObservedRunningTime="2026-09-14 01:41:05.51086079 +0000 UTC m=+610.744226049" Sep 14 01:41:05.525321 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:05.525276 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" podStartSLOduration=1.525261022 podStartE2EDuration="1.525261022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:05.524679628 +0000 UTC m=+610.758044903" watchObservedRunningTime="2026-09-14 01:41:05.525261022 +0000 UTC m=+610.758626279" Sep 14 01:41:05.526628 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:05.526601 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-x7tlk\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="2e3e40e4-28a2-4f15-9858-6d0b0a6345ee" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-x7tlk" Sep 14 01:41:07.045501 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:07.045463 2579 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-gx686_kuadrant-system_17f34a68-6960-454c-a266-5eb19992f660_0(e8794853a40689b2eab55f256cef3b51a16e12982b1f0b1bb0ca84ade3eb1f88): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-gx686 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Sep 14 01:41:07.045752 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:07.045525 2579 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-gx686_kuadrant-system_17f34a68-6960-454c-a266-5eb19992f660_0(e8794853a40689b2eab55f256cef3b51a16e12982b1f0b1bb0ca84ade3eb1f88): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-gx686 to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:07.450751 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.450674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-sbs6s" Sep 14 01:41:07.501517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.501487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:07.509049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.509023 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:07.511383 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:07.511351 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="17f34a68-6960-454c-a266-5eb19992f660" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:07.540940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.540911 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl76g\" (UniqueName: \"kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g\") pod \"17f34a68-6960-454c-a266-5eb19992f660\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " Sep 14 01:41:07.541129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.540965 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume\") pod \"17f34a68-6960-454c-a266-5eb19992f660\" (UID: \"17f34a68-6960-454c-a266-5eb19992f660\") " Sep 14 01:41:07.541259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.541230 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "17f34a68-6960-454c-a266-5eb19992f660" (UID: "17f34a68-6960-454c-a266-5eb19992f660"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:41:07.542881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.542861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g" (OuterVolumeSpecName: "kube-api-access-wl76g") pod "17f34a68-6960-454c-a266-5eb19992f660" (UID: "17f34a68-6960-454c-a266-5eb19992f660"). InnerVolumeSpecName "kube-api-access-wl76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:07.641769 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.641741 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl76g\" (UniqueName: \"kubernetes.io/projected/17f34a68-6960-454c-a266-5eb19992f660-kube-api-access-wl76g\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:07.641769 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.641763 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f34a68-6960-454c-a266-5eb19992f660-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:07.694259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.694226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:07.786647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:07.786597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:08.505769 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:08.505736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:08.507779 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:08.507748 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="17f34a68-6960-454c-a266-5eb19992f660" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:08.513314 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:08.513291 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-gx686\" is forbidden: User \"system:node:ip-10-0-130-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-123.ec2.internal' and this object" podUID="17f34a68-6960-454c-a266-5eb19992f660" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-gx686" Sep 14 01:41:09.209712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:09.209681 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f34a68-6960-454c-a266-5eb19992f660" path="/var/lib/kubelet/pods/17f34a68-6960-454c-a266-5eb19992f660/volumes" Sep 14 01:41:12.485430 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:12.485384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:12.551001 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:12.550961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:16.499169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.499142 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:16.499686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.499220 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:16.563303 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.563270 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:16.564127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.563489 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" podUID="d747025a-ee60-4587-bba1-fa886b67db53" containerName="manager" containerID="cri-o://d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6" gracePeriod=10 Sep 14 01:41:16.780144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.780108 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 01:41:16.783694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.783673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.804458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.804435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 01:41:16.811322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.811301 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:16.821437 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.821411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.821576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.821452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhpt\" (UniqueName: \"kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.922312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82h8m\" (UniqueName: \"kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m\") pod \"d747025a-ee60-4587-bba1-fa886b67db53\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " Sep 14 01:41:16.922469 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume\") pod \"d747025a-ee60-4587-bba1-fa886b67db53\" (UID: \"d747025a-ee60-4587-bba1-fa886b67db53\") " Sep 14 01:41:16.922510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.922510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhpt\" (UniqueName: \"kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.922714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d747025a-ee60-4587-bba1-fa886b67db53" (UID: "d747025a-ee60-4587-bba1-fa886b67db53"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:41:16.922884 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.922865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:16.924287 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.924266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m" (OuterVolumeSpecName: "kube-api-access-82h8m") pod "d747025a-ee60-4587-bba1-fa886b67db53" (UID: "d747025a-ee60-4587-bba1-fa886b67db53"). InnerVolumeSpecName "kube-api-access-82h8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:16.937826 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:16.937799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhpt\" (UniqueName: \"kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mkt6k\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:17.023155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.023091 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82h8m\" (UniqueName: \"kubernetes.io/projected/d747025a-ee60-4587-bba1-fa886b67db53-kube-api-access-82h8m\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:17.023155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.023117 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d747025a-ee60-4587-bba1-fa886b67db53-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:17.107539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.107507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:17.392391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.392363 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 01:41:17.394919 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:41:17.394890 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc227b5_9c96_4db8_9683_8c3e883deb83.slice/crio-93b6b4a329fe869baf4ec646765b2e188e59536c9ea15e074b63812d8e7d418d WatchSource:0}: Error finding container 93b6b4a329fe869baf4ec646765b2e188e59536c9ea15e074b63812d8e7d418d: Status 404 returned error can't find the container with id 93b6b4a329fe869baf4ec646765b2e188e59536c9ea15e074b63812d8e7d418d Sep 14 01:41:17.552475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.552444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" event={"ID":"4dc227b5-9c96-4db8-9683-8c3e883deb83","Type":"ContainerStarted","Data":"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf"} Sep 14 01:41:17.552828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.552509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:17.552828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.552536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" event={"ID":"4dc227b5-9c96-4db8-9683-8c3e883deb83","Type":"ContainerStarted","Data":"93b6b4a329fe869baf4ec646765b2e188e59536c9ea15e074b63812d8e7d418d"} Sep 14 01:41:17.553861 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.553834 2579 generic.go:356] "Generic (PLEG): container finished" podID="d747025a-ee60-4587-bba1-fa886b67db53" containerID="d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6" exitCode=0 Sep 14 01:41:17.553934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.553874 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" Sep 14 01:41:17.553976 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.553874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" event={"ID":"d747025a-ee60-4587-bba1-fa886b67db53","Type":"ContainerDied","Data":"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6"} Sep 14 01:41:17.554021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.553995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv" event={"ID":"d747025a-ee60-4587-bba1-fa886b67db53","Type":"ContainerDied","Data":"0afb3eed82963401719aa9dc78606bc87651e4c818393eaa907386360b4ffa0b"} Sep 14 01:41:17.554021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.554017 2579 scope.go:117] "RemoveContainer" containerID="d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6" Sep 14 01:41:17.563942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.563926 2579 scope.go:117] "RemoveContainer" containerID="d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6" Sep 14 01:41:17.564262 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:17.564243 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6\": container with ID starting with d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6 not found: ID does not exist" containerID="d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6" Sep 14 01:41:17.564360 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.564273 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6"} err="failed to get container status \"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6\": rpc error: code = NotFound desc = could not find container \"d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6\": container with ID starting with d41cde26fb1699b678a93cbca9a930fab48f93f17ed45c2a2e7949da265ba1b6 not found: ID does not exist" Sep 14 01:41:17.579993 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.579952 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" podStartSLOduration=1.579940474 podStartE2EDuration="1.579940474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:17.578171813 +0000 UTC m=+622.811537091" watchObservedRunningTime="2026-09-14 01:41:17.579940474 +0000 UTC m=+622.813305730" Sep 14 01:41:17.596998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.596977 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:17.611761 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.611742 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dsbxv"] Sep 14 01:41:17.704167 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.704086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:17.808003 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:17.807949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:18.086524 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:18.086494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:18.185425 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:18.185396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:18.290845 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:18.290814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:18.392125 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:18.392004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:19.209568 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:19.209536 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d747025a-ee60-4587-bba1-fa886b67db53" path="/var/lib/kubelet/pods/d747025a-ee60-4587-bba1-fa886b67db53/volumes" Sep 14 01:41:24.726160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:24.726122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:24.827369 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:24.827323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:28.561742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.561706 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 01:41:28.608873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.608840 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:28.609050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.609030 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" podUID="960281cf-f6ee-434e-836b-84d261f93766" containerName="manager" containerID="cri-o://6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b" gracePeriod=10 Sep 14 01:41:28.851466 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.851439 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:28.924706 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.924679 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4sqv\" (UniqueName: \"kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv\") pod \"960281cf-f6ee-434e-836b-84d261f93766\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " Sep 14 01:41:28.924706 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.924714 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume\") pod \"960281cf-f6ee-434e-836b-84d261f93766\" (UID: \"960281cf-f6ee-434e-836b-84d261f93766\") " Sep 14 01:41:28.925137 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.925110 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "960281cf-f6ee-434e-836b-84d261f93766" (UID: "960281cf-f6ee-434e-836b-84d261f93766"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:41:28.926676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:28.926655 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv" (OuterVolumeSpecName: "kube-api-access-v4sqv") pod "960281cf-f6ee-434e-836b-84d261f93766" (UID: "960281cf-f6ee-434e-836b-84d261f93766"). InnerVolumeSpecName "kube-api-access-v4sqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:29.025881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.025845 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4sqv\" (UniqueName: \"kubernetes.io/projected/960281cf-f6ee-434e-836b-84d261f93766-kube-api-access-v4sqv\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:29.025881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.025876 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/960281cf-f6ee-434e-836b-84d261f93766-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:41:29.489683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.489643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:29.605229 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.605197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:29.612322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.612299 2579 generic.go:356] "Generic (PLEG): container finished" podID="960281cf-f6ee-434e-836b-84d261f93766" containerID="6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b" exitCode=0 Sep 14 01:41:29.612473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.612338 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" Sep 14 01:41:29.612473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.612383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" event={"ID":"960281cf-f6ee-434e-836b-84d261f93766","Type":"ContainerDied","Data":"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b"} Sep 14 01:41:29.612473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.612442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm" event={"ID":"960281cf-f6ee-434e-836b-84d261f93766","Type":"ContainerDied","Data":"e1dac0ff1990266e910f0f79f28a2f9c10553311ec8eecad65d1204817ef1f1e"} Sep 14 01:41:29.612473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.612472 2579 scope.go:117] "RemoveContainer" containerID="6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b" Sep 14 01:41:29.621020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.621002 2579 scope.go:117] "RemoveContainer" containerID="6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b" Sep 14 01:41:29.621296 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:41:29.621278 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b\": container with ID starting with 6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b not found: ID does not exist" containerID="6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b" Sep 14 01:41:29.621340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.621305 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b"} err="failed to get container status \"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b\": rpc error: code = NotFound desc = could not find container \"6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b\": container with ID starting with 6d45fea4fe9231f821697583e4326e141b9c8869b8254981789c26502610300b not found: ID does not exist" Sep 14 01:41:29.639997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.639976 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:29.648122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:29.648050 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ln4mm"] Sep 14 01:41:30.103662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:30.103623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:30.204085 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:30.204028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:31.209342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:31.209310 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="960281cf-f6ee-434e-836b-84d261f93766" path="/var/lib/kubelet/pods/960281cf-f6ee-434e-836b-84d261f93766/volumes" Sep 14 01:41:32.872737 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:32.872705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:33.192699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:33.192618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:34.282804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:34.282773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:34.383264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:34.383220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:41.909374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:41.909340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:42.013209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:42.013164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:46.239763 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:46.239722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:46.348562 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:46.348535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:46.449387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:46.449343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:47.109340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.109304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:47.188686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.188652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:41:47.192654 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.192629 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:41:47.193001 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.192989 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960281cf-f6ee-434e-836b-84d261f93766" containerName="manager" Sep 14 01:41:47.193048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.193003 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="960281cf-f6ee-434e-836b-84d261f93766" containerName="manager" Sep 14 01:41:47.193048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.193020 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d747025a-ee60-4587-bba1-fa886b67db53" containerName="manager" Sep 14 01:41:47.193048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.193025 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d747025a-ee60-4587-bba1-fa886b67db53" containerName="manager" Sep 14 01:41:47.193171 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.193126 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="960281cf-f6ee-434e-836b-84d261f93766" containerName="manager" Sep 14 01:41:47.193171 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.193138 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d747025a-ee60-4587-bba1-fa886b67db53" containerName="manager" Sep 14 01:41:47.197202 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.197179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:41:47.199374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.199357 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gr8lh\"" Sep 14 01:41:47.202516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.202494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:41:47.284570 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.284528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgqg\" (UniqueName: \"kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg\") pod \"authorino-59c974bd48-vdfz4\" (UID: \"c2ddfda6-b815-442d-8632-4b100b74228a\") " pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:41:47.385470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.385402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgqg\" (UniqueName: \"kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg\") pod \"authorino-59c974bd48-vdfz4\" (UID: \"c2ddfda6-b815-442d-8632-4b100b74228a\") " pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:41:47.393722 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.393697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgqg\" (UniqueName: \"kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg\") pod \"authorino-59c974bd48-vdfz4\" (UID: \"c2ddfda6-b815-442d-8632-4b100b74228a\") " pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:41:47.507171 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.507141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:41:47.776518 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:47.776389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:41:47.778913 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:41:47.778887 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ddfda6_b815_442d_8632_4b100b74228a.slice/crio-d0e4d50c641b491bb23be10b51b9405127cccddf109e432aaa7882cd1fe5ff8f WatchSource:0}: Error finding container d0e4d50c641b491bb23be10b51b9405127cccddf109e432aaa7882cd1fe5ff8f: Status 404 returned error can't find the container with id d0e4d50c641b491bb23be10b51b9405127cccddf109e432aaa7882cd1fe5ff8f Sep 14 01:41:48.706310 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:48.706278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vdfz4" event={"ID":"c2ddfda6-b815-442d-8632-4b100b74228a","Type":"ContainerStarted","Data":"d0e4d50c641b491bb23be10b51b9405127cccddf109e432aaa7882cd1fe5ff8f"} Sep 14 01:41:50.717515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:50.717424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vdfz4" event={"ID":"c2ddfda6-b815-442d-8632-4b100b74228a","Type":"ContainerStarted","Data":"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80"} Sep 14 01:41:50.731739 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:41:50.731690 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-vdfz4" podStartSLOduration=1.157783512 podStartE2EDuration="3.731676331s" totalImagesPullingTime="2.573892819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:50.730218669 +0000 UTC m=+655.963583940" watchObservedRunningTime="2026-09-14 01:41:50.731676331 +0000 UTC m=+655.965041588" Sep 14 01:42:03.629263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:03.629225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:03.724472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:03.724438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:04.083554 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:04.083513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:06.333019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:06.332978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:06.434983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:06.434953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:15.828820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:15.828785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:15.997041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:15.996999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:16.290906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.290872 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:42:16.297250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.297223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.299552 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.299528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 14 01:42:16.300031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.300006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:42:16.338899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.338861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp95\" (UniqueName: \"kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.339048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.338912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.439890 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.439856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp95\" (UniqueName: \"kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.440084 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.439908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.442316 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.442296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.446648 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.446631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp95\" (UniqueName: \"kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95\") pod \"authorino-79bcdb5ddd-ddwc8\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.608356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.608274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:42:16.875705 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.875679 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:42:16.877778 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:42:16.877749 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8469a1e_fb58_42ba_9c22_8b1197ff23f2.slice/crio-2985c8af9970820109e45632364ae4d74328286ad3621e3d0efd3cbcb1ec9c98 WatchSource:0}: Error finding container 2985c8af9970820109e45632364ae4d74328286ad3621e3d0efd3cbcb1ec9c98: Status 404 returned error can't find the container with id 2985c8af9970820109e45632364ae4d74328286ad3621e3d0efd3cbcb1ec9c98 Sep 14 01:42:16.879321 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:16.879303 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:42:17.842843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:17.842757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" event={"ID":"f8469a1e-fb58-42ba-9c22-8b1197ff23f2","Type":"ContainerStarted","Data":"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d"} Sep 14 01:42:17.842843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:17.842805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" event={"ID":"f8469a1e-fb58-42ba-9c22-8b1197ff23f2","Type":"ContainerStarted","Data":"2985c8af9970820109e45632364ae4d74328286ad3621e3d0efd3cbcb1ec9c98"} Sep 14 01:42:17.857257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:17.857212 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" podStartSLOduration=1.191821547 podStartE2EDuration="1.857196859s" totalImagesPullingTime="665.375312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:42:17.855599437 +0000 UTC m=+683.088964696" watchObservedRunningTime="2026-09-14 01:42:17.857196859 +0000 UTC m=+683.090562117" Sep 14 01:42:17.882204 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:17.882176 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:42:17.882611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:17.882339 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-vdfz4" podUID="c2ddfda6-b815-442d-8632-4b100b74228a" containerName="authorino" containerID="cri-o://fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80" gracePeriod=30 Sep 14 01:42:18.122447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.122426 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:42:18.254440 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.254408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsgqg\" (UniqueName: \"kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg\") pod \"c2ddfda6-b815-442d-8632-4b100b74228a\" (UID: \"c2ddfda6-b815-442d-8632-4b100b74228a\") " Sep 14 01:42:18.256412 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.256386 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg" (OuterVolumeSpecName: "kube-api-access-dsgqg") pod "c2ddfda6-b815-442d-8632-4b100b74228a" (UID: "c2ddfda6-b815-442d-8632-4b100b74228a"). InnerVolumeSpecName "kube-api-access-dsgqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:42:18.356098 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.356031 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgqg\" (UniqueName: \"kubernetes.io/projected/c2ddfda6-b815-442d-8632-4b100b74228a-kube-api-access-dsgqg\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:42:18.848817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.848723 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2ddfda6-b815-442d-8632-4b100b74228a" containerID="fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80" exitCode=0 Sep 14 01:42:18.848817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.848783 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vdfz4" Sep 14 01:42:18.849145 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.848812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vdfz4" event={"ID":"c2ddfda6-b815-442d-8632-4b100b74228a","Type":"ContainerDied","Data":"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80"} Sep 14 01:42:18.849145 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.848852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vdfz4" event={"ID":"c2ddfda6-b815-442d-8632-4b100b74228a","Type":"ContainerDied","Data":"d0e4d50c641b491bb23be10b51b9405127cccddf109e432aaa7882cd1fe5ff8f"} Sep 14 01:42:18.849145 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.848869 2579 scope.go:117] "RemoveContainer" containerID="fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80" Sep 14 01:42:18.857762 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.857747 2579 scope.go:117] "RemoveContainer" containerID="fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80" Sep 14 01:42:18.858026 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:42:18.858008 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80\": container with ID starting with fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80 not found: ID does not exist" containerID="fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80" Sep 14 01:42:18.858094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.858031 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80"} err="failed to get container status \"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80\": rpc error: code = NotFound desc = could not find container \"fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80\": container with ID starting with fab1c1600448e36c8620630c0c2d9c89ae2398cbe3ec51212753a2f6316bfc80 not found: ID does not exist" Sep 14 01:42:18.869546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.869521 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:42:18.872790 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:18.872772 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vdfz4"] Sep 14 01:42:19.209582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:19.209500 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ddfda6-b815-442d-8632-4b100b74228a" path="/var/lib/kubelet/pods/c2ddfda6-b815-442d-8632-4b100b74228a/volumes" Sep 14 01:42:22.521469 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:22.521433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:22.589732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:22.589684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:22.639197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:22.639166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:23.272445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:23.272416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:26.664315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:26.664277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:26.779346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:26.779309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:28.137588 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:28.137554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:28.237698 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:28.237662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:58.051301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:58.051263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:58.150829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:58.150793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:59.441610 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:59.441572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:59.541553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:59.541515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:59.710239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:59.710154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:59.811088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:59.811047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:42:59.916941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:42:59.916901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:00.006557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.006517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:00.026628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.026595 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd"] Sep 14 01:43:00.026958 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.026945 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ddfda6-b815-442d-8632-4b100b74228a" containerName="authorino" Sep 14 01:43:00.027015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.026960 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ddfda6-b815-442d-8632-4b100b74228a" containerName="authorino" Sep 14 01:43:00.027134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.027015 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ddfda6-b815-442d-8632-4b100b74228a" containerName="authorino" Sep 14 01:43:00.031104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.031052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.034672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.034656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dwfhr\"" Sep 14 01:43:00.035549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.035533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 01:43:00.046534 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.046506 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd"] Sep 14 01:43:00.114473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.114435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e8bd73-3af1-471c-b362-427bfdcbf437-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.114625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.114593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hs6d\" (UniqueName: \"kubernetes.io/projected/62e8bd73-3af1-471c-b362-427bfdcbf437-kube-api-access-7hs6d\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.215073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.215027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e8bd73-3af1-471c-b362-427bfdcbf437-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.215232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.215150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hs6d\" (UniqueName: \"kubernetes.io/projected/62e8bd73-3af1-471c-b362-427bfdcbf437-kube-api-access-7hs6d\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.215963 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.215938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62e8bd73-3af1-471c-b362-427bfdcbf437-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.223645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.223615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hs6d\" (UniqueName: \"kubernetes.io/projected/62e8bd73-3af1-471c-b362-427bfdcbf437-kube-api-access-7hs6d\") pod \"payload-pre-processing-59fbc47bf8-9twnd\" (UID: \"62e8bd73-3af1-471c-b362-427bfdcbf437\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.342217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.342138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:00.611120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:00.610985 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd"] Sep 14 01:43:00.613487 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:43:00.613462 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e8bd73_3af1_471c_b362_427bfdcbf437.slice/crio-ed469ff575c00b696984eef1956e9ea8301022bd083665eab36aae8a6079cb9a WatchSource:0}: Error finding container ed469ff575c00b696984eef1956e9ea8301022bd083665eab36aae8a6079cb9a: Status 404 returned error can't find the container with id ed469ff575c00b696984eef1956e9ea8301022bd083665eab36aae8a6079cb9a Sep 14 01:43:01.053012 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:01.052978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" event={"ID":"62e8bd73-3af1-471c-b362-427bfdcbf437","Type":"ContainerStarted","Data":"ed469ff575c00b696984eef1956e9ea8301022bd083665eab36aae8a6079cb9a"} Sep 14 01:43:02.059348 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:02.059314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" event={"ID":"62e8bd73-3af1-471c-b362-427bfdcbf437","Type":"ContainerStarted","Data":"1964e06766b150741ee9da999c7208e79fd388a0c17e7593accd9748af6b9192"} Sep 14 01:43:02.059799 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:02.059372 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:02.075577 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:02.075487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" podStartSLOduration=1.866127075 podStartE2EDuration="3.075471691s" totalImagesPullingTime="1.209344616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:02.074045634 +0000 UTC m=+727.307410891" watchObservedRunningTime="2026-09-14 01:43:02.075471691 +0000 UTC m=+727.308836947" Sep 14 01:43:11.906212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:11.906170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:11.996896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:11.996852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:13.066499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:13.066465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9twnd" Sep 14 01:43:17.017109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.017036 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599"] Sep 14 01:43:17.020511 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.020492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.023929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.023900 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 01:43:17.024047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.023960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 14 01:43:17.024047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.023912 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 01:43:17.024047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.023986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tpvrt\"" Sep 14 01:43:17.030738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.030713 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599"] Sep 14 01:43:17.047746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.047720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:17.166662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97394f1-ef43-4919-8a9d-118cb991dddb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.166662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.166849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.166849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.166849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lvs\" (UniqueName: \"kubernetes.io/projected/e97394f1-ef43-4919-8a9d-118cb991dddb-kube-api-access-59lvs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.166849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.166821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267317 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lvs\" (UniqueName: \"kubernetes.io/projected/e97394f1-ef43-4919-8a9d-118cb991dddb-kube-api-access-59lvs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267317 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97394f1-ef43-4919-8a9d-118cb991dddb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267837 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.267901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.267876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.269943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.269919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e97394f1-ef43-4919-8a9d-118cb991dddb-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.270045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.269918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e97394f1-ef43-4919-8a9d-118cb991dddb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.274951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.274931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lvs\" (UniqueName: \"kubernetes.io/projected/e97394f1-ef43-4919-8a9d-118cb991dddb-kube-api-access-59lvs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599\" (UID: \"e97394f1-ef43-4919-8a9d-118cb991dddb\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.335379 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.335350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:17.610248 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:17.610169 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599"] Sep 14 01:43:17.614226 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:43:17.614184 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97394f1_ef43_4919_8a9d_118cb991dddb.slice/crio-bf3b7dd8339567067a501e5b5a167868d0077b23aec13c1cd0a5a6bc7e28d1ae WatchSource:0}: Error finding container bf3b7dd8339567067a501e5b5a167868d0077b23aec13c1cd0a5a6bc7e28d1ae: Status 404 returned error can't find the container with id bf3b7dd8339567067a501e5b5a167868d0077b23aec13c1cd0a5a6bc7e28d1ae Sep 14 01:43:18.139463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:18.139429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" event={"ID":"e97394f1-ef43-4919-8a9d-118cb991dddb","Type":"ContainerStarted","Data":"bf3b7dd8339567067a501e5b5a167868d0077b23aec13c1cd0a5a6bc7e28d1ae"} Sep 14 01:43:19.146021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:19.145928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" event={"ID":"e97394f1-ef43-4919-8a9d-118cb991dddb","Type":"ContainerStarted","Data":"3d5b6ea36b37a4fe385eafd3d4b51be94b9db2a0c646acedeb61abfcaf6d45fe"} Sep 14 01:43:19.163707 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:19.163643 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" podStartSLOduration=2.008793927 podStartE2EDuration="3.163626595s" totalImagesPullingTime="1.154832668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:19.162660558 +0000 UTC m=+744.396025839" watchObservedRunningTime="2026-09-14 01:43:19.163626595 +0000 UTC m=+744.396991851" Sep 14 01:43:25.211700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.211664 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5"] Sep 14 01:43:25.215489 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.215469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.219110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.219054 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 14 01:43:25.220722 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.220693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5"] Sep 14 01:43:25.237682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.237657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:25.348388 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.348556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jf4\" (UniqueName: \"kubernetes.io/projected/66c6d581-1156-4101-9860-32ebfe300c5c-kube-api-access-c4jf4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.348556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.348696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.348696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.348696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.348636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6d581-1156-4101-9860-32ebfe300c5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jf4\" (UniqueName: \"kubernetes.io/projected/66c6d581-1156-4101-9860-32ebfe300c5c-kube-api-access-c4jf4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6d581-1156-4101-9860-32ebfe300c5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449914 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.449960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.449948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.451973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.451946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c6d581-1156-4101-9860-32ebfe300c5c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.452325 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.452306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c6d581-1156-4101-9860-32ebfe300c5c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.457903 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.457877 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jf4\" (UniqueName: \"kubernetes.io/projected/66c6d581-1156-4101-9860-32ebfe300c5c-kube-api-access-c4jf4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xpnn5\" (UID: \"66c6d581-1156-4101-9860-32ebfe300c5c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.529050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.529014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:25.798201 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:25.798172 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5"] Sep 14 01:43:25.800896 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:43:25.800867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c6d581_1156_4101_9860_32ebfe300c5c.slice/crio-5441fe735df813090c769ec28957aaacd74bdd131d543f4f7ddf59846fd940fb WatchSource:0}: Error finding container 5441fe735df813090c769ec28957aaacd74bdd131d543f4f7ddf59846fd940fb: Status 404 returned error can't find the container with id 5441fe735df813090c769ec28957aaacd74bdd131d543f4f7ddf59846fd940fb Sep 14 01:43:26.179553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:26.179514 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" event={"ID":"66c6d581-1156-4101-9860-32ebfe300c5c","Type":"ContainerStarted","Data":"a9fa8d0f961b0667e04af4d086bf621b4159be8feb3076fa42b4c75ba1a36aa2"} Sep 14 01:43:26.179752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:26.179581 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" event={"ID":"66c6d581-1156-4101-9860-32ebfe300c5c","Type":"ContainerStarted","Data":"5441fe735df813090c769ec28957aaacd74bdd131d543f4f7ddf59846fd940fb"} Sep 14 01:43:26.199254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:26.199201 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" podStartSLOduration=0.978681677 podStartE2EDuration="1.199184787s" totalImagesPullingTime="220.50311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:26.198310333 +0000 UTC m=+751.431675611" watchObservedRunningTime="2026-09-14 01:43:26.199184787 +0000 UTC m=+751.432550044" Sep 14 01:43:27.335522 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:27.335490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:27.335873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:27.335541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:27.347976 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:27.347953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:28.200174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:28.200146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599" Sep 14 01:43:35.529160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:35.529120 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:35.529556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:35.529180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:35.541806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:35.541783 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:36.236645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:36.236617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xpnn5" Sep 14 01:43:45.040837 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:45.040799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:43:56.921404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:43:56.921374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:44:13.342115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:44:13.342078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:44:18.038398 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:44:18.038314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:44:31.703043 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:44:31.703006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:00.137502 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.137463 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j"] Sep 14 01:45:00.140997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.140974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" Sep 14 01:45:00.143281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.143260 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:45:00.146158 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.146140 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j"] Sep 14 01:45:00.151952 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.151930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" Sep 14 01:45:00.422661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.422608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j"] Sep 14 01:45:00.424958 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:45:00.424921 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5427eb6b_c262_4452_ae79_9c3be2919231.slice/crio-e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9 WatchSource:0}: Error finding container e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9: Status 404 returned error can't find the container with id e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9 Sep 14 01:45:00.624582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:00.624545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" event={"ID":"5427eb6b-c262-4452-ae79-9c3be2919231","Type":"ContainerStarted","Data":"e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9"} Sep 14 01:45:04.645688 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:04.645652 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" event={"ID":"5427eb6b-c262-4452-ae79-9c3be2919231","Type":"ContainerStarted","Data":"a2ef750788e7573c143e4ffc2d1f8687df9331921b0b9ba483fb50ae31be91cd"} Sep 14 01:45:04.660116 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:04.660069 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" podStartSLOduration=1.126823627 podStartE2EDuration="4.660039859s" totalImagesPullingTime="3.533216232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:45:04.658509996 +0000 UTC m=+849.891875276" watchObservedRunningTime="2026-09-14 01:45:04.660039859 +0000 UTC m=+849.893405116" Sep 14 01:45:05.652127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:05.652092 2579 generic.go:356] "Generic (PLEG): container finished" podID="5427eb6b-c262-4452-ae79-9c3be2919231" containerID="a2ef750788e7573c143e4ffc2d1f8687df9331921b0b9ba483fb50ae31be91cd" exitCode=0 Sep 14 01:45:05.652514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:05.652181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" event={"ID":"5427eb6b-c262-4452-ae79-9c3be2919231","Type":"ContainerDied","Data":"a2ef750788e7573c143e4ffc2d1f8687df9331921b0b9ba483fb50ae31be91cd"} Sep 14 01:45:06.783697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:06.783675 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" Sep 14 01:45:07.663911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:07.663886 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" Sep 14 01:45:07.664091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:07.663884 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822505-gnq2j" event={"ID":"5427eb6b-c262-4452-ae79-9c3be2919231","Type":"ContainerDied","Data":"e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9"} Sep 14 01:45:07.664091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:07.664015 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50fcab4d9576726d230ff5bb7327ead82d4dbb7f68ea3d88750b521f3c780a9" Sep 14 01:45:18.807889 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:18.807856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:19.220112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:19.220009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:19.271330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:19.271304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:45.168045 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.168000 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:45:45.168632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.168309 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" podUID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" containerName="authorino" containerID="cri-o://ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d" gracePeriod=30 Sep 14 01:45:45.424078 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.423986 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:45:45.565980 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.565947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert\") pod \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " Sep 14 01:45:45.566165 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.565993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxp95\" (UniqueName: \"kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95\") pod \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\" (UID: \"f8469a1e-fb58-42ba-9c22-8b1197ff23f2\") " Sep 14 01:45:45.567990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.567956 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95" (OuterVolumeSpecName: "kube-api-access-lxp95") pod "f8469a1e-fb58-42ba-9c22-8b1197ff23f2" (UID: "f8469a1e-fb58-42ba-9c22-8b1197ff23f2"). InnerVolumeSpecName "kube-api-access-lxp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:45:45.575852 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.575825 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f8469a1e-fb58-42ba-9c22-8b1197ff23f2" (UID: "f8469a1e-fb58-42ba-9c22-8b1197ff23f2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:45:45.666911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.666872 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-tls-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:45:45.666911 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.666909 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxp95\" (UniqueName: \"kubernetes.io/projected/f8469a1e-fb58-42ba-9c22-8b1197ff23f2-kube-api-access-lxp95\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:45:45.693484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.693403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:45.780015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.779968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:45.855143 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.855108 2579 generic.go:356] "Generic (PLEG): container finished" podID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" containerID="ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d" exitCode=0 Sep 14 01:45:45.855329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.855155 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" Sep 14 01:45:45.855329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.855193 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" event={"ID":"f8469a1e-fb58-42ba-9c22-8b1197ff23f2","Type":"ContainerDied","Data":"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d"} Sep 14 01:45:45.855329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.855235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bcdb5ddd-ddwc8" event={"ID":"f8469a1e-fb58-42ba-9c22-8b1197ff23f2","Type":"ContainerDied","Data":"2985c8af9970820109e45632364ae4d74328286ad3621e3d0efd3cbcb1ec9c98"} Sep 14 01:45:45.855329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.855257 2579 scope.go:117] "RemoveContainer" containerID="ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d" Sep 14 01:45:45.864629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.864611 2579 scope.go:117] "RemoveContainer" containerID="ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d" Sep 14 01:45:45.864885 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:45:45.864867 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d\": container with ID starting with ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d not found: ID does not exist" containerID="ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d" Sep 14 01:45:45.864932 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.864894 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d"} err="failed to get container status \"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d\": rpc error: code = NotFound desc = could not find container \"ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d\": container with ID starting with ad35044b4b531a78110fdce502615d096152b97a87c6c0137abf0be90f441d8d not found: ID does not exist" Sep 14 01:45:45.875783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.875762 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:45:45.879263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:45.879241 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79bcdb5ddd-ddwc8"] Sep 14 01:45:46.875391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:46.875351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:46.969683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:46.969634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:45:47.210536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:47.210456 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" path="/var/lib/kubelet/pods/f8469a1e-fb58-42ba-9c22-8b1197ff23f2/volumes" Sep 14 01:45:55.195933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:55.195903 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:45:55.197968 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:45:55.197945 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:46:04.117662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:04.117622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:05.105556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.105521 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 01:46:05.106162 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106142 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5427eb6b-c262-4452-ae79-9c3be2919231" containerName="cleanup" Sep 14 01:46:05.106162 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106163 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5427eb6b-c262-4452-ae79-9c3be2919231" containerName="cleanup" Sep 14 01:46:05.106336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106200 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" containerName="authorino" Sep 14 01:46:05.106336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106209 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" containerName="authorino" Sep 14 01:46:05.106336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106299 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8469a1e-fb58-42ba-9c22-8b1197ff23f2" containerName="authorino" Sep 14 01:46:05.106336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.106312 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5427eb6b-c262-4452-ae79-9c3be2919231" containerName="cleanup" Sep 14 01:46:05.109781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.109762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.112138 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.112113 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8e5935-openshift-default-dockercfg-xkmc2\"" Sep 14 01:46:05.120929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.120908 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 01:46:05.167093 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.166731 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:46:05.170780 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.170758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.173139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.173118 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-99c616-openshift-default-dockercfg-v7f8t\"" Sep 14 01:46:05.181430 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.181405 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:46:05.240342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvtv\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.240677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.240665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341146 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341333 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341333 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341333 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341333 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvtv\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341498 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341498 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341498 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.341785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.341948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zh7\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.342163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.342163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.341973 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.342163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.342018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.342163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.342096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.342293 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.342239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.343865 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.343838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.343865 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.343845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.348621 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.348599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.349098 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.349078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvtv\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv\") pod \"e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.365108 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.365018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:05.420910 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.420884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:05.443230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zh7\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443805 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.443859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.444008 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.443985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.444276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.444249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.445922 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.445895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.446244 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.446223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.450834 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.450811 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.451049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.451029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zh7\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7\") pod \"e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.465445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.465413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:05.482785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.482457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:05.566619 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.566582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:05.694925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.694900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 01:46:05.696168 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:05.696147 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2088d45_5b19_4cf9_a333_978ceb60d588.slice/crio-e1370bd6aa18e404fb89a579cf9a587400bbe975817cdc1d8d8056176814b5d9 WatchSource:0}: Error finding container e1370bd6aa18e404fb89a579cf9a587400bbe975817cdc1d8d8056176814b5d9: Status 404 returned error can't find the container with id e1370bd6aa18e404fb89a579cf9a587400bbe975817cdc1d8d8056176814b5d9 Sep 14 01:46:05.698326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.698295 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.698382 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.698358 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.698421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.698385 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.762234 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.762161 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:46:05.764323 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:05.764293 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23fc97e_d029_416a_a4f9_e1789b960950.slice/crio-17ae18275fc8203f1f125ed15a1f5750794ae06f8d96a81f9c50d42fe3ca2569 WatchSource:0}: Error finding container 17ae18275fc8203f1f125ed15a1f5750794ae06f8d96a81f9c50d42fe3ca2569: Status 404 returned error can't find the container with id 17ae18275fc8203f1f125ed15a1f5750794ae06f8d96a81f9c50d42fe3ca2569 Sep 14 01:46:05.766852 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.766823 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.766931 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.766888 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.766971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.766936 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:05.958543 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.958460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerStarted","Data":"3493a436bc1c94678c2f00bf0bb9d88768e6ca812db17b36fed2801140304ae7"} Sep 14 01:46:05.958770 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.958749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerStarted","Data":"17ae18275fc8203f1f125ed15a1f5750794ae06f8d96a81f9c50d42fe3ca2569"} Sep 14 01:46:05.960361 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.960327 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerStarted","Data":"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416"} Sep 14 01:46:05.960682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.960373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerStarted","Data":"e1370bd6aa18e404fb89a579cf9a587400bbe975817cdc1d8d8056176814b5d9"} Sep 14 01:46:05.976629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.976578 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podStartSLOduration=0.976558357 podStartE2EDuration="976.558357ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:05.975854066 +0000 UTC m=+911.209219350" watchObservedRunningTime="2026-09-14 01:46:05.976558357 +0000 UTC m=+911.209923616" Sep 14 01:46:05.997438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:05.997382 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podStartSLOduration=0.997361128 podStartE2EDuration="997.361128ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:05.99530626 +0000 UTC m=+911.228671569" watchObservedRunningTime="2026-09-14 01:46:05.997361128 +0000 UTC m=+911.230726387" Sep 14 01:46:06.207969 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.207928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:06.307599 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.307564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:06.421461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.421426 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:06.427190 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.427162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:06.483259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.483226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:06.488493 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.488465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:06.749935 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.749892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:06.965163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.965121 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:06.965346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.965210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:06.966003 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.965982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:06.966123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:06.966045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:07.973598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:07.973566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:08.067416 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:08.067372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:08.166751 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:08.166720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:08.584052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:08.584017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:08.680985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:08.680947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:09.706809 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:09.706775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:09.804038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:09.803999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:09.904386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:09.904351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:10.337941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.337909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:10.439741 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.439707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:10.444700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.444670 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:46:10.448239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.448221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.450555 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.450533 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-99c616\"" Sep 14 01:46:10.450648 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.450624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-99c616-dockercfg-4psnv\"" Sep 14 01:46:10.458995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.458906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:46:10.469110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.469083 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:46:10.472400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.472376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.482420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.482395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:46:10.491298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.491272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrcz\" (UniqueName: \"kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.491417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.491331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.491417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.491356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.491502 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.491480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqgf\" (UniqueName: \"kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.591963 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.591884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.591963 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.591917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.592197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.591997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqgf\" (UniqueName: \"kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.592197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.592054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrcz\" (UniqueName: \"kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.592700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.592680 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.592770 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.592752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.599604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.599581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqgf\" (UniqueName: \"kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf\") pod \"payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.599695 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.599632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrcz\" (UniqueName: \"kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz\") pod \"payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.759706 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.759669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:10.786088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.786036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:10.861560 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.861523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:10.960338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:10.960294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:11.036889 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.036755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:46:11.041390 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:11.041366 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eddaf66_7848_4f73_9ca0_1470091cae38.slice/crio-b36faf631b4d69c231eda4114ccd021d0dde2a4f5e8da91b1a04332a90315e46 WatchSource:0}: Error finding container b36faf631b4d69c231eda4114ccd021d0dde2a4f5e8da91b1a04332a90315e46: Status 404 returned error can't find the container with id b36faf631b4d69c231eda4114ccd021d0dde2a4f5e8da91b1a04332a90315e46 Sep 14 01:46:11.074387 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.074365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:46:11.076136 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:11.076109 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24f59e4_36ec_4d26_af2c_55cbadffa802.slice/crio-df6f1efacacc453aed8cbae53506319d89ef08fe51e9bc5c862e56f040ab5f6d WatchSource:0}: Error finding container df6f1efacacc453aed8cbae53506319d89ef08fe51e9bc5c862e56f040ab5f6d: Status 404 returned error can't find the container with id df6f1efacacc453aed8cbae53506319d89ef08fe51e9bc5c862e56f040ab5f6d Sep 14 01:46:11.289936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.289890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:11.413960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.413871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:11.515219 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.515180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:11.525449 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.525419 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 01:46:11.529170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.529147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.531883 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.531856 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8e5935-dockercfg-9f954\"" Sep 14 01:46:11.532009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.531862 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8e5935\"" Sep 14 01:46:11.540795 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.540772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 01:46:11.557198 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.557172 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 01:46:11.560901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.560884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.570700 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.570680 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 01:46:11.601054 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.601024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknvq\" (UniqueName: \"kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.601198 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.601177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.702585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.702495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknvq\" (UniqueName: \"kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.702585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.702556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.702808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.702623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.702808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.702646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pps\" (UniqueName: \"kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.703275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.703255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.710309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.710287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknvq\" (UniqueName: \"kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq\") pod \"payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.803418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.803381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.803418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.803421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pps\" (UniqueName: \"kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.804029 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.804009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.812017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.811990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pps\" (UniqueName: \"kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps\") pod \"payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.841747 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.841724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:11.871687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.871661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:11.928423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.928047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:11.992813 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.992739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" event={"ID":"9eddaf66-7848-4f73-9ca0-1470091cae38","Type":"ContainerStarted","Data":"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7"} Sep 14 01:46:11.992813 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.992806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:11.993040 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.992833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" event={"ID":"9eddaf66-7848-4f73-9ca0-1470091cae38","Type":"ContainerStarted","Data":"b36faf631b4d69c231eda4114ccd021d0dde2a4f5e8da91b1a04332a90315e46"} Sep 14 01:46:11.994464 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.994432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" event={"ID":"a24f59e4-36ec-4d26-af2c-55cbadffa802","Type":"ContainerStarted","Data":"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c"} Sep 14 01:46:11.994598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.994491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:11.994598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:11.994515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" event={"ID":"a24f59e4-36ec-4d26-af2c-55cbadffa802","Type":"ContainerStarted","Data":"df6f1efacacc453aed8cbae53506319d89ef08fe51e9bc5c862e56f040ab5f6d"} Sep 14 01:46:12.029550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.029497 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" podStartSLOduration=2.029481014 podStartE2EDuration="2.029481014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:12.010004847 +0000 UTC m=+917.243370132" watchObservedRunningTime="2026-09-14 01:46:12.029481014 +0000 UTC m=+917.262846316" Sep 14 01:46:12.029725 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.029581 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" podStartSLOduration=2.029577127 podStartE2EDuration="2.029577127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:12.027771942 +0000 UTC m=+917.261137218" watchObservedRunningTime="2026-09-14 01:46:12.029577127 +0000 UTC m=+917.262942385" Sep 14 01:46:12.114535 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.114511 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 01:46:12.116277 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:12.116248 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38688eb8_d5c1_4594_aca2_6c676f3b61dc.slice/crio-74f13cc75704e0fc56c870c22f367e91f96a5090d96fe238ba7941633855b01d WatchSource:0}: Error finding container 74f13cc75704e0fc56c870c22f367e91f96a5090d96fe238ba7941633855b01d: Status 404 returned error can't find the container with id 74f13cc75704e0fc56c870c22f367e91f96a5090d96fe238ba7941633855b01d Sep 14 01:46:12.163962 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.163938 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 01:46:12.165583 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:12.165560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277ca8a3_1391_4c51_89cb_95590ea55b71.slice/crio-d33b9b9ec8597cf97740988d1f54101f40e69b641fb0a5126cf6f8c87424a0e5 WatchSource:0}: Error finding container d33b9b9ec8597cf97740988d1f54101f40e69b641fb0a5126cf6f8c87424a0e5: Status 404 returned error can't find the container with id d33b9b9ec8597cf97740988d1f54101f40e69b641fb0a5126cf6f8c87424a0e5 Sep 14 01:46:12.854601 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.854567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:12.909524 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:12.909468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.001409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.001370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" event={"ID":"38688eb8-d5c1-4594-aca2-6c676f3b61dc","Type":"ContainerStarted","Data":"4e8b50e703862d92588242479e2192e9f491630467d7d3de9878c144bb1243ee"} Sep 14 01:46:13.001584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.001438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" event={"ID":"38688eb8-d5c1-4594-aca2-6c676f3b61dc","Type":"ContainerStarted","Data":"74f13cc75704e0fc56c870c22f367e91f96a5090d96fe238ba7941633855b01d"} Sep 14 01:46:13.001584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.001467 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:13.003028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.003000 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" event={"ID":"277ca8a3-1391-4c51-89cb-95590ea55b71","Type":"ContainerStarted","Data":"3ab25ac81a45d0fd17e7ed0c1e85d6d9bb2f39d721966d3369d0be339cb57578"} Sep 14 01:46:13.003189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.003050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" event={"ID":"277ca8a3-1391-4c51-89cb-95590ea55b71","Type":"ContainerStarted","Data":"d33b9b9ec8597cf97740988d1f54101f40e69b641fb0a5126cf6f8c87424a0e5"} Sep 14 01:46:13.003189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.003107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:13.012174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.012142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.018817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.018765 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" podStartSLOduration=2.018746207 podStartE2EDuration="2.018746207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:13.016053663 +0000 UTC m=+918.249418938" watchObservedRunningTime="2026-09-14 01:46:13.018746207 +0000 UTC m=+918.252111466" Sep 14 01:46:13.033315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.033261 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" podStartSLOduration=2.03324294 podStartE2EDuration="2.03324294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:13.031700289 +0000 UTC m=+918.265065573" watchObservedRunningTime="2026-09-14 01:46:13.03324294 +0000 UTC m=+918.266608229" Sep 14 01:46:13.114281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.114180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.409656 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.409532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.419697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.419666 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:13.425549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.425523 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.428119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.428090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e3d76dcd-openshift-default-dockercfg-rlk7k\"" Sep 14 01:46:13.438008 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.437914 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:13.504765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.504725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.520941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.520905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.520970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wd2\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.521421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.521363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.540541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.540499 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:46:13.544227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.544204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.546620 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.546597 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a36bf3-dockercfg-4nbs7\"" Sep 14 01:46:13.546754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.546639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a36bf3\"" Sep 14 01:46:13.555797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.555771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:46:13.604948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.604917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.622070 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wd2\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.622590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcl86\" (UniqueName: \"kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.622590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.622990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.622968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.623285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.623261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.623466 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.623280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.623785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.623762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.624760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.624740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.625347 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.625326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.629615 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.629591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wd2\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.629748 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.629729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token\") pod \"e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.724205 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.724118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.724205 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.724186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcl86\" (UniqueName: \"kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.724827 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.724804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.733248 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.733224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcl86\" (UniqueName: \"kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86\") pod \"payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:13.742900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.742877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:13.839897 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.839864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:13.858520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:13.858495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:14.021828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.021803 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:14.023049 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:14.023024 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda070084c_a411_4869_be23_86b3b9db6828.slice/crio-b9a9586e18d3cfc78ae9f06880addfd0a6e418ec5582a22df890936cd4fd52f0 WatchSource:0}: Error finding container b9a9586e18d3cfc78ae9f06880addfd0a6e418ec5582a22df890936cd4fd52f0: Status 404 returned error can't find the container with id b9a9586e18d3cfc78ae9f06880addfd0a6e418ec5582a22df890936cd4fd52f0 Sep 14 01:46:14.025256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.025228 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:14.025356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.025285 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:14.025356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.025318 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:14.141727 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.141697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:46:14.143153 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:14.143112 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba63483_50fb_446e_bbdf_fc621b0f044b.slice/crio-f4e2f70571ad80f2f0929f73fd20b198d03f9f22c160abafa7d09f150b224f2e WatchSource:0}: Error finding container f4e2f70571ad80f2f0929f73fd20b198d03f9f22c160abafa7d09f150b224f2e: Status 404 returned error can't find the container with id f4e2f70571ad80f2f0929f73fd20b198d03f9f22c160abafa7d09f150b224f2e Sep 14 01:46:14.459757 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.459670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:14.560404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.560362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:14.782168 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.782133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:14.851818 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:14.851777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:15.016104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.016044 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerStarted","Data":"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29"} Sep 14 01:46:15.016570 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.016136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerStarted","Data":"b9a9586e18d3cfc78ae9f06880addfd0a6e418ec5582a22df890936cd4fd52f0"} Sep 14 01:46:15.017816 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.017789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" event={"ID":"fba63483-50fb-446e-bbdf-fc621b0f044b","Type":"ContainerStarted","Data":"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42"} Sep 14 01:46:15.017964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.017833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" event={"ID":"fba63483-50fb-446e-bbdf-fc621b0f044b","Type":"ContainerStarted","Data":"f4e2f70571ad80f2f0929f73fd20b198d03f9f22c160abafa7d09f150b224f2e"} Sep 14 01:46:15.017964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.017884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:15.036780 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.036693 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podStartSLOduration=2.036681024 podStartE2EDuration="2.036681024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:15.03454176 +0000 UTC m=+920.267907036" watchObservedRunningTime="2026-09-14 01:46:15.036681024 +0000 UTC m=+920.270046281" Sep 14 01:46:15.051859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.051813 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" podStartSLOduration=2.051797497 podStartE2EDuration="2.051797497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:15.051272312 +0000 UTC m=+920.284637598" watchObservedRunningTime="2026-09-14 01:46:15.051797497 +0000 UTC m=+920.285162755" Sep 14 01:46:15.334595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.334509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:15.743243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.743208 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:15.748497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:15.748464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:16.024020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:16.023923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:16.024733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:16.024712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:16.047385 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:16.047348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:16.481837 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:16.481806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:16.581230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:16.581192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:17.250518 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:17.250480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:17.784422 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:17.784388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:17.874680 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:17.874647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:18.613174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:18.613135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:19.401478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:19.401447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.171886 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.171852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.242196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.242159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.461138 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.461024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.582699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.582647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.683775 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.683746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:20.721178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.718169 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:20.745292 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.743975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:20.745292 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.744028 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:20.751110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.746299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.751469 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.751446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e3d76dcd\"" Sep 14 01:46:20.751890 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.751676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e3d76dcd-dockercfg-xhvdp\"" Sep 14 01:46:20.756666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.756647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:20.756874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.756860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.793608 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.793581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs992\" (UniqueName: \"kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.793747 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.793726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4gv\" (UniqueName: \"kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.793812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.793790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.793871 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.793824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.894852 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.894823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs992\" (UniqueName: \"kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.895156 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.895130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4gv\" (UniqueName: \"kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.895329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.895310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.895475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.895460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.896023 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.895999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.896250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.896199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:20.903132 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.903115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4gv\" (UniqueName: \"kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv\") pod \"payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:20.903243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:20.903172 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs992\" (UniqueName: \"kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992\") pod \"payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:21.062299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.062263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:21.084541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.084500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:21.090105 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.090082 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:21.159011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.158935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:21.275089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.268596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:21.437811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.437728 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:21.455748 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.455098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:21.459887 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:21.459179 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af96ba7_8f66_4645_9faf_e7510a6f8d39.slice/crio-c1c19c884b39dfa0f95a6765a31653c87d63fbe13d8724b98bce08f0c8648f6c WatchSource:0}: Error finding container c1c19c884b39dfa0f95a6765a31653c87d63fbe13d8724b98bce08f0c8648f6c: Status 404 returned error can't find the container with id c1c19c884b39dfa0f95a6765a31653c87d63fbe13d8724b98bce08f0c8648f6c Sep 14 01:46:21.466615 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:21.466569 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f22b88_f070_421c_9acf_06bc6c71d0d8.slice/crio-d6aef1226c63bee9c890fdc27bbf9578ac16c94e76d1a78db3f33c8924d691d8 WatchSource:0}: Error finding container d6aef1226c63bee9c890fdc27bbf9578ac16c94e76d1a78db3f33c8924d691d8: Status 404 returned error can't find the container with id d6aef1226c63bee9c890fdc27bbf9578ac16c94e76d1a78db3f33c8924d691d8 Sep 14 01:46:21.557705 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.557665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:21.623082 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:21.621410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:22.065346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.065305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:22.075704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.075664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" event={"ID":"3af96ba7-8f66-4645-9faf-e7510a6f8d39","Type":"ContainerStarted","Data":"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f"} Sep 14 01:46:22.075881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.075732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" event={"ID":"3af96ba7-8f66-4645-9faf-e7510a6f8d39","Type":"ContainerStarted","Data":"c1c19c884b39dfa0f95a6765a31653c87d63fbe13d8724b98bce08f0c8648f6c"} Sep 14 01:46:22.075975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.075952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:22.078417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.078383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" event={"ID":"70f22b88-f070-421c-9acf-06bc6c71d0d8","Type":"ContainerStarted","Data":"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686"} Sep 14 01:46:22.078556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.078436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" event={"ID":"70f22b88-f070-421c-9acf-06bc6c71d0d8","Type":"ContainerStarted","Data":"d6aef1226c63bee9c890fdc27bbf9578ac16c94e76d1a78db3f33c8924d691d8"} Sep 14 01:46:22.078658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.078637 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:22.092838 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.092795 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" podStartSLOduration=2.092782033 podStartE2EDuration="2.092782033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:22.091110712 +0000 UTC m=+927.324475989" watchObservedRunningTime="2026-09-14 01:46:22.092782033 +0000 UTC m=+927.326147289" Sep 14 01:46:22.107258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.107208 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" podStartSLOduration=2.107191232 podStartE2EDuration="2.107191232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:22.104361823 +0000 UTC m=+927.337727092" watchObservedRunningTime="2026-09-14 01:46:22.107191232 +0000 UTC m=+927.340556491" Sep 14 01:46:22.770000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.769942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:22.885494 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:22.885452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.006286 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.006248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:46:23.007219 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.007194 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:46:23.296208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.296165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.384077 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.384025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.487752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.487719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.692221 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.692101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.795257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.794906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:23.952306 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:23.952188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:24.011731 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:24.010695 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 01:46:24.011937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:24.011911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 01:46:24.032681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:24.032638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:24.099764 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:24.099733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:24.803344 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:24.803300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:25.593627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:25.593589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:26.026112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:26.026080 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:46:26.437743 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:26.437607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:26.934153 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:26.934092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:27.118349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.116500 2579 generic.go:356] "Generic (PLEG): container finished" podID="a23fc97e-d029-416a-a4f9-e1789b960950" containerID="3493a436bc1c94678c2f00bf0bb9d88768e6ca812db17b36fed2801140304ae7" exitCode=0 Sep 14 01:46:27.118349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.116650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerDied","Data":"3493a436bc1c94678c2f00bf0bb9d88768e6ca812db17b36fed2801140304ae7"} Sep 14 01:46:27.118349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.117191 2579 scope.go:117] "RemoveContainer" containerID="3493a436bc1c94678c2f00bf0bb9d88768e6ca812db17b36fed2801140304ae7" Sep 14 01:46:27.120015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.119968 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:27.120148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.120086 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:27.120148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.120139 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:27.175984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.175919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:27.877359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:27.877316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:28.084317 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.084282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:28.129253 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.129146 2579 generic.go:356] "Generic (PLEG): container finished" podID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerID="cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416" exitCode=0 Sep 14 01:46:28.129253 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.129230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerDied","Data":"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416"} Sep 14 01:46:28.129821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.129798 2579 scope.go:117] "RemoveContainer" containerID="cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416" Sep 14 01:46:28.131357 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.131320 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:28.131585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.131561 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:28.131743 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.131724 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:28.140305 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.140243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerStarted","Data":"e22af9638cf91eb536f25090c76b28c90dde716cdde3c1604b3cd00d00b93490"} Sep 14 01:46:28.359906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.359745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:28.461870 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.461783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:28.483534 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.483504 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:28.483831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.483537 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:28.483989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.483955 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:28.559886 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.559848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:28.676283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:28.676220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:29.155741 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.155698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerStarted","Data":"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663"} Sep 14 01:46:29.211406 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.209781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:29.422370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.422260 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:29.422370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.422310 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:29.422613 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.422371 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:29.483669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.483620 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:29.483840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:29.483682 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:30.169328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.169206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:30.386597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.386560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:30.496486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.495481 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:30.496486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.495536 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:30.496486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.495623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:30.496486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.496185 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:30.496486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.496237 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:30.612391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:30.612353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:31.150192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:31.150104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:31.421576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:31.421477 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:31.422173 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:31.422137 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:31.483894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:31.483849 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:31.484122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:31.483911 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:32.083174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.083136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:32.421455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.421410 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:32.421657 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.421478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:32.483180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.483138 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:32.483364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.483200 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:32.977420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:32.977383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:33.087716 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.087687 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:33.087904 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.087771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:33.421772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.421735 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:33.422335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.422299 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:33.483404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.483364 2579 patch_prober.go:28] interesting pod/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 14 01:46:33.483638 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.483612 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 14 01:46:33.651644 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.651618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:33.752548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:33.752507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:34.026909 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.026771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:34.323130 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.320352 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:46:34.329084 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.328435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.336207 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.334683 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:46:34.336207 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.335007 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-321811-openshift-default-dockercfg-gjqtq\"" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347148 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f5j\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.347949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.347603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.349808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.349775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:34.421905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.421425 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 14 01:46:34.421905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.421500 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 14 01:46:34.449642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.449833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.449833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.449833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.449833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f5j\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.450097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.450097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.450097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.449926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.450097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.450003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.451154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.450761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.451154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.450769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.451154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.451014 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.451154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.451086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.451383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.451334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.453024 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.452979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.453696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.453673 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.457516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.457470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.459018 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.458989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f5j\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j\") pod \"e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.487169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.487136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:34.487781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.487565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:34.488245 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.488230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:46:34.655085 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.654979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:34.665021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.664992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:34.766629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.766578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:34.965865 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.965837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:46:34.968634 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:34.968606 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfba7ba_4d93_4b93_a7e0_c092f576ed80.slice/crio-bc5afadd49dcee5d2290ced684e42eafaed0a9197c0c03ebed234d03dfcb70d1 WatchSource:0}: Error finding container bc5afadd49dcee5d2290ced684e42eafaed0a9197c0c03ebed234d03dfcb70d1: Status 404 returned error can't find the container with id bc5afadd49dcee5d2290ced684e42eafaed0a9197c0c03ebed234d03dfcb70d1 Sep 14 01:46:34.971836 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.971790 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:34.971964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.971893 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:34.971964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:34.971934 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:35.213228 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.213194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerStarted","Data":"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a"} Sep 14 01:46:35.213394 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.213254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerStarted","Data":"bc5afadd49dcee5d2290ced684e42eafaed0a9197c0c03ebed234d03dfcb70d1"} Sep 14 01:46:35.229579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.229525 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podStartSLOduration=1.229505176 podStartE2EDuration="1.229505176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:35.227129545 +0000 UTC m=+940.460494833" watchObservedRunningTime="2026-09-14 01:46:35.229505176 +0000 UTC m=+940.462870437" Sep 14 01:46:35.425260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.425230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:35.425616 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.425589 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:35.426397 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.426377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 01:46:35.553730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.553568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:35.656177 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:35.656143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:36.292446 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.292413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:36.395544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.395511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:36.495223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.495188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:36.658943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.658842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:36.661104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.661083 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:36.852537 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.852500 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx"] Sep 14 01:46:36.855013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.854996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:36.857283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.857261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:46:36.857409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.857335 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:46:36.863381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.863357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx"] Sep 14 01:46:36.877461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.877434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx\" (UID: \"d1231de3-2148-4b9b-b74f-ed449231edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:36.979014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.978980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx\" (UID: \"d1231de3-2148-4b9b-b74f-ed449231edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:36.979581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:36.979563 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx\" (UID: \"d1231de3-2148-4b9b-b74f-ed449231edb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:37.141989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.141958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:37.166899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.166867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:37.227030 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.227005 2579 generic.go:356] "Generic (PLEG): container finished" podID="a070084c-a411-4869-be23-86b3b9db6828" containerID="0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29" exitCode=0 Sep 14 01:46:37.227265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.227119 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerDied","Data":"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29"} Sep 14 01:46:37.227840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.227813 2579 scope.go:117] "RemoveContainer" containerID="0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29" Sep 14 01:46:37.228504 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.228279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:37.230855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.230364 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:37.230855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.230450 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:37.230855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.230491 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:37.231052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.230985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:37.243244 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.243198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:37.342366 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.342328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:37.442686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.442642 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx"] Sep 14 01:46:37.446659 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:37.446615 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1231de3_2148_4b9b_b74f_ed449231edb1.slice/crio-4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc WatchSource:0}: Error finding container 4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc: Status 404 returned error can't find the container with id 4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc Sep 14 01:46:37.514091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.514039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:37.614462 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:37.614423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:38.237439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.237402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" event={"ID":"d1231de3-2148-4b9b-b74f-ed449231edb1","Type":"ContainerStarted","Data":"4fda18b9c7d4154ae8b0a1e59e094a6ac3fd8dd75878345fbeb7d156cedd7e8c"} Sep 14 01:46:38.237644 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.237465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" event={"ID":"d1231de3-2148-4b9b-b74f-ed449231edb1","Type":"ContainerStarted","Data":"4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc"} Sep 14 01:46:38.239568 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.239545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerStarted","Data":"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74"} Sep 14 01:46:38.252978 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.252930 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" podStartSLOduration=2.2529171630000002 podStartE2EDuration="2.252917163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:38.251396968 +0000 UTC m=+943.484762247" watchObservedRunningTime="2026-09-14 01:46:38.252917163 +0000 UTC m=+943.486282419" Sep 14 01:46:38.295402 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.295360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:38.743458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.743426 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:38.743822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.743612 2579 patch_prober.go:28] interesting pod/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:46:38.743822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:38.743658 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 14 01:46:39.067456 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.067353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:39.245928 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.245891 2579 generic.go:356] "Generic (PLEG): container finished" podID="d1231de3-2148-4b9b-b74f-ed449231edb1" containerID="4fda18b9c7d4154ae8b0a1e59e094a6ac3fd8dd75878345fbeb7d156cedd7e8c" exitCode=0 Sep 14 01:46:39.246141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.245971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" event={"ID":"d1231de3-2148-4b9b-b74f-ed449231edb1","Type":"ContainerDied","Data":"4fda18b9c7d4154ae8b0a1e59e094a6ac3fd8dd75878345fbeb7d156cedd7e8c"} Sep 14 01:46:39.743559 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.743519 2579 patch_prober.go:28] interesting pod/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:46:39.743962 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.743591 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 14 01:46:39.779194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.779158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:39.879979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.879940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:39.980547 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:39.980513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:40.250703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.250668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:40.351088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.350896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:40.389123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.389092 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:46:40.392841 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.392818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.399927 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.399894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-321811\"" Sep 14 01:46:40.401034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.401011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-321811-dockercfg-n59v8\"" Sep 14 01:46:40.403893 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.403871 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:46:40.445590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.445549 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:40.535981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.535883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca\") pod \"d1231de3-2148-4b9b-b74f-ed449231edb1\" (UID: \"d1231de3-2148-4b9b-b74f-ed449231edb1\") " Sep 14 01:46:40.536285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.536257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6qn\" (UniqueName: \"kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.536383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.536312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.536435 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.536402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1231de3-2148-4b9b-b74f-ed449231edb1" (UID: "d1231de3-2148-4b9b-b74f-ed449231edb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:46:40.637760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.637726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6qn\" (UniqueName: \"kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.637760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.637773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.638035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.637886 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1231de3-2148-4b9b-b74f-ed449231edb1-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:40.638584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.638557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.645031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.645009 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6qn\" (UniqueName: \"kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn\") pod \"payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.662579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.662547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:40.741445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.741408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:40.744211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.743967 2579 patch_prober.go:28] interesting pod/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:46:40.744211 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:40.744016 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 14 01:46:41.016762 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.016719 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:46:41.262080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.262026 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" Sep 14 01:46:41.262080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.262023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx" event={"ID":"d1231de3-2148-4b9b-b74f-ed449231edb1","Type":"ContainerDied","Data":"4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc"} Sep 14 01:46:41.262306 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.262132 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4892b8471eaec9bf415e35e86930cee54d03b20bba758a671bd6fe5b57fa9ddc" Sep 14 01:46:41.263934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.263868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" event={"ID":"b228d39c-ba77-4a57-9da4-39070f186b6f","Type":"ContainerStarted","Data":"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906"} Sep 14 01:46:41.263934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.263925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" event={"ID":"b228d39c-ba77-4a57-9da4-39070f186b6f","Type":"ContainerStarted","Data":"d96784d038d7f71c1b3667e934471f24ad008717e4e403b8849fc552e80ed859"} Sep 14 01:46:41.264156 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.263956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:41.280319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.280273 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" podStartSLOduration=1.280259399 podStartE2EDuration="1.280259399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:41.278667809 +0000 UTC m=+946.512033089" watchObservedRunningTime="2026-09-14 01:46:41.280259399 +0000 UTC m=+946.513624655" Sep 14 01:46:41.487921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.487888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:41.744017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.743924 2579 patch_prober.go:28] interesting pod/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:46:41.744017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:41.743996 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 14 01:46:42.223335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.223223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:42.589947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.589911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:42.690303 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.690265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:42.743579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.743545 2579 patch_prober.go:28] interesting pod/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:46:42.743717 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.743613 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 14 01:46:42.964631 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:42.964541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:43.727308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:43.727272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:43.748038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:43.748007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:43.748486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:43.748467 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:43.749073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:43.749042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:44.341195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:44.341161 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:46.300403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:46.300373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.297183 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.297076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.322123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.322091 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:47.322593 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.322312 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" podUID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" containerName="payload-processing" containerID="cri-o://8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f" gracePeriod=30 Sep 14 01:46:47.330359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.330332 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:47.331894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.331850 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" podUID="70f22b88-f070-421c-9acf-06bc6c71d0d8" containerName="payload-pre-processing" containerID="cri-o://43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686" gracePeriod=30 Sep 14 01:46:47.361305 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.361267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.426269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.426221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.509010 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.508962 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx"] Sep 14 01:46:47.514524 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.513379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.517619 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.517576 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3d76dcd-ccgzx"] Sep 14 01:46:47.579673 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.579518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.628579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.628531 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:47.628806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.628779 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" containerID="cri-o://d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74" gracePeriod=30 Sep 14 01:46:47.659906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.659882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:47.671943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.671920 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:47.675273 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.675255 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:47.815704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.815664 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume\") pod \"70f22b88-f070-421c-9acf-06bc6c71d0d8\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " Sep 14 01:46:47.815899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.815764 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume\") pod \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " Sep 14 01:46:47.815899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.815882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq4gv\" (UniqueName: \"kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv\") pod \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\" (UID: \"3af96ba7-8f66-4645-9faf-e7510a6f8d39\") " Sep 14 01:46:47.816019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.815927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs992\" (UniqueName: \"kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992\") pod \"70f22b88-f070-421c-9acf-06bc6c71d0d8\" (UID: \"70f22b88-f070-421c-9acf-06bc6c71d0d8\") " Sep 14 01:46:47.816140 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.816112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70f22b88-f070-421c-9acf-06bc6c71d0d8" (UID: "70f22b88-f070-421c-9acf-06bc6c71d0d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:46:47.816204 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.816143 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3af96ba7-8f66-4645-9faf-e7510a6f8d39" (UID: "3af96ba7-8f66-4645-9faf-e7510a6f8d39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:46:47.816288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.816265 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af96ba7-8f66-4645-9faf-e7510a6f8d39-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:47.816288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.816285 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70f22b88-f070-421c-9acf-06bc6c71d0d8-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:47.818194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.818163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv" (OuterVolumeSpecName: "kube-api-access-kq4gv") pod "3af96ba7-8f66-4645-9faf-e7510a6f8d39" (UID: "3af96ba7-8f66-4645-9faf-e7510a6f8d39"). InnerVolumeSpecName "kube-api-access-kq4gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:46:47.818272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.818203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992" (OuterVolumeSpecName: "kube-api-access-gs992") pod "70f22b88-f070-421c-9acf-06bc6c71d0d8" (UID: "70f22b88-f070-421c-9acf-06bc6c71d0d8"). InnerVolumeSpecName "kube-api-access-gs992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:46:47.917192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.917113 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq4gv\" (UniqueName: \"kubernetes.io/projected/3af96ba7-8f66-4645-9faf-e7510a6f8d39-kube-api-access-kq4gv\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:47.917192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.917141 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs992\" (UniqueName: \"kubernetes.io/projected/70f22b88-f070-421c-9acf-06bc6c71d0d8-kube-api-access-gs992\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:47.944696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:47.944656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.107325 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.107286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.208032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.207953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.308178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.308144 2579 generic.go:356] "Generic (PLEG): container finished" podID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" containerID="8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f" exitCode=0 Sep 14 01:46:48.308404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.308192 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" Sep 14 01:46:48.308404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.308239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" event={"ID":"3af96ba7-8f66-4645-9faf-e7510a6f8d39","Type":"ContainerDied","Data":"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f"} Sep 14 01:46:48.308404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.308294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr" event={"ID":"3af96ba7-8f66-4645-9faf-e7510a6f8d39","Type":"ContainerDied","Data":"c1c19c884b39dfa0f95a6765a31653c87d63fbe13d8724b98bce08f0c8648f6c"} Sep 14 01:46:48.308404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.308316 2579 scope.go:117] "RemoveContainer" containerID="8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f" Sep 14 01:46:48.309591 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.309569 2579 generic.go:356] "Generic (PLEG): container finished" podID="70f22b88-f070-421c-9acf-06bc6c71d0d8" containerID="43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686" exitCode=0 Sep 14 01:46:48.309756 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.309596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" event={"ID":"70f22b88-f070-421c-9acf-06bc6c71d0d8","Type":"ContainerDied","Data":"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686"} Sep 14 01:46:48.309756 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.309623 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" Sep 14 01:46:48.309756 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.309656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj" event={"ID":"70f22b88-f070-421c-9acf-06bc6c71d0d8","Type":"ContainerDied","Data":"d6aef1226c63bee9c890fdc27bbf9578ac16c94e76d1a78db3f33c8924d691d8"} Sep 14 01:46:48.318353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.318338 2579 scope.go:117] "RemoveContainer" containerID="8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f" Sep 14 01:46:48.318631 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:46:48.318612 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f\": container with ID starting with 8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f not found: ID does not exist" containerID="8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f" Sep 14 01:46:48.318721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.318643 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f"} err="failed to get container status \"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f\": rpc error: code = NotFound desc = could not find container \"8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f\": container with ID starting with 8d62dd454410cac80b7b637f0d7ff6adba96792f394384972b095687d9b9ab0f not found: ID does not exist" Sep 14 01:46:48.318721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.318666 2579 scope.go:117] "RemoveContainer" containerID="43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686" Sep 14 01:46:48.328441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.328409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.333038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.333012 2579 scope.go:117] "RemoveContainer" containerID="43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686" Sep 14 01:46:48.333415 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:46:48.333381 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686\": container with ID starting with 43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686 not found: ID does not exist" containerID="43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686" Sep 14 01:46:48.333511 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.333420 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686"} err="failed to get container status \"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686\": rpc error: code = NotFound desc = could not find container \"43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686\": container with ID starting with 43a89afe14b28423896a9edf2541e753855ffe1a0b48a2a09557791449dae686 not found: ID does not exist" Sep 14 01:46:48.335472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.335446 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:48.340651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.340623 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e3d76dcd-6bcc677f4b-cp9bj"] Sep 14 01:46:48.355687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.355645 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:48.360229 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.360198 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e3d76dcd-ff7c5dd59-rj8gr"] Sep 14 01:46:48.599470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.599433 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:46:48.600129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600106 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f22b88-f070-421c-9acf-06bc6c71d0d8" containerName="payload-pre-processing" Sep 14 01:46:48.600129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600128 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f22b88-f070-421c-9acf-06bc6c71d0d8" containerName="payload-pre-processing" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600147 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" containerName="payload-processing" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600155 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" containerName="payload-processing" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600176 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1231de3-2148-4b9b-b74f-ed449231edb1" containerName="revoke-keys" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600184 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1231de3-2148-4b9b-b74f-ed449231edb1" containerName="revoke-keys" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600287 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" containerName="payload-processing" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600297 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f22b88-f070-421c-9acf-06bc6c71d0d8" containerName="payload-pre-processing" Sep 14 01:46:48.600318 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.600304 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1231de3-2148-4b9b-b74f-ed449231edb1" containerName="revoke-keys" Sep 14 01:46:48.602647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.602629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.605368 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.605352 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b766dd7c-gw-openshift-default-dockercfg-64q9v\"" Sep 14 01:46:48.614472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.614448 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:46:48.635945 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.635913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.725526 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwr9\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.725947 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.725934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.786679 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.786646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:48.826893 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.826858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827026 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.826914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827026 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.826963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827026 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.826985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwr9\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827593 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.827776 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.827764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.829768 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.829737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.830006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.829989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.834354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.834334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwr9\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.834439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.834367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token\") pod \"e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.914664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.914582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:48.955336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:48.955302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:49.187420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.187393 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:46:49.188621 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:49.188597 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5acee9_f465_487c_828f_78d9a4d78917.slice/crio-282f8bfc6a1b520d406d2b2afaca1da19a99931fe94cb27d6b119bb0c13f8495 WatchSource:0}: Error finding container 282f8bfc6a1b520d406d2b2afaca1da19a99931fe94cb27d6b119bb0c13f8495: Status 404 returned error can't find the container with id 282f8bfc6a1b520d406d2b2afaca1da19a99931fe94cb27d6b119bb0c13f8495 Sep 14 01:46:49.190914 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.190886 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:49.191032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.190946 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:49.191032 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.190974 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:49.212784 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.212761 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af96ba7-8f66-4645-9faf-e7510a6f8d39" path="/var/lib/kubelet/pods/3af96ba7-8f66-4645-9faf-e7510a6f8d39/volumes" Sep 14 01:46:49.213131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.213117 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f22b88-f070-421c-9acf-06bc6c71d0d8" path="/var/lib/kubelet/pods/70f22b88-f070-421c-9acf-06bc6c71d0d8/volumes" Sep 14 01:46:49.213429 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.213418 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1231de3-2148-4b9b-b74f-ed449231edb1" path="/var/lib/kubelet/pods/d1231de3-2148-4b9b-b74f-ed449231edb1/volumes" Sep 14 01:46:49.319876 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.319832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerStarted","Data":"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b"} Sep 14 01:46:49.320077 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.319903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerStarted","Data":"282f8bfc6a1b520d406d2b2afaca1da19a99931fe94cb27d6b119bb0c13f8495"} Sep 14 01:46:49.341859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.341802 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podStartSLOduration=1.3417777370000001 podStartE2EDuration="1.341777737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:49.340382134 +0000 UTC m=+954.573747391" watchObservedRunningTime="2026-09-14 01:46:49.341777737 +0000 UTC m=+954.575142996" Sep 14 01:46:49.647128 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.647090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:49.915231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:49.915036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:50.433427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:50.433387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:50.925264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:50.925226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:50.942793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:50.942758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:51.043617 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:51.043574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:51.234080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:51.232390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:51.338013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:51.337971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:51.338987 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:51.338964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:46:51.923714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:51.923677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:52.026051 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:52.026012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:52.271557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:52.271528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:46:52.647900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:52.647850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:52.922146 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:52.906675 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:53.072033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.071997 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072166 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072211 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9wd2\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072362 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072404 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:46:53.072444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072439 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072465 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072503 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket\") pod \"a070084c-a411-4869-be23-86b3b9db6828\" (UID: \"a070084c-a411-4869-be23-86b3b9db6828\") " Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072530 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data" (OuterVolumeSpecName: "istio-data") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072603 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072737 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:46:53.072882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072883 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.073239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072894 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.073239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072904 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a070084c-a411-4869-be23-86b3b9db6828-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.073239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072912 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.073239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.072919 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.074509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.074483 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:46:53.074727 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.074704 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:46:53.074801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.074778 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token" (OuterVolumeSpecName: "istio-token") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:46:53.074841 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.074810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2" (OuterVolumeSpecName: "kube-api-access-d9wd2") pod "a070084c-a411-4869-be23-86b3b9db6828" (UID: "a070084c-a411-4869-be23-86b3b9db6828"). InnerVolumeSpecName "kube-api-access-d9wd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:46:53.174531 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.174448 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9wd2\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-kube-api-access-d9wd2\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.174531 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.174479 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a070084c-a411-4869-be23-86b3b9db6828-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.174531 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.174488 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a070084c-a411-4869-be23-86b3b9db6828-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.174531 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.174496 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a070084c-a411-4869-be23-86b3b9db6828-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:46:53.324193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.324155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:53.354585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.354552 2579 generic.go:356] "Generic (PLEG): container finished" podID="a070084c-a411-4869-be23-86b3b9db6828" containerID="d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74" exitCode=0 Sep 14 01:46:53.354765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.354610 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" Sep 14 01:46:53.354765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.354632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerDied","Data":"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74"} Sep 14 01:46:53.354765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.354723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn" event={"ID":"a070084c-a411-4869-be23-86b3b9db6828","Type":"ContainerDied","Data":"b9a9586e18d3cfc78ae9f06880addfd0a6e418ec5582a22df890936cd4fd52f0"} Sep 14 01:46:53.354765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.354749 2579 scope.go:117] "RemoveContainer" containerID="d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74" Sep 14 01:46:53.364090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.364072 2579 scope.go:117] "RemoveContainer" containerID="0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29" Sep 14 01:46:53.372072 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.372046 2579 scope.go:117] "RemoveContainer" containerID="d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74" Sep 14 01:46:53.372377 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:46:53.372356 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74\": container with ID starting with d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74 not found: ID does not exist" containerID="d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74" Sep 14 01:46:53.372423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.372385 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74"} err="failed to get container status \"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74\": rpc error: code = NotFound desc = could not find container \"d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74\": container with ID starting with d559ee2e10868bc057fc2e88a0d4ad0947f6cb10843d48f5e4063d8a91dbab74 not found: ID does not exist" Sep 14 01:46:53.372423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.372401 2579 scope.go:117] "RemoveContainer" containerID="0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29" Sep 14 01:46:53.372660 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:46:53.372643 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29\": container with ID starting with 0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29 not found: ID does not exist" containerID="0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29" Sep 14 01:46:53.372704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.372666 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29"} err="failed to get container status \"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29\": rpc error: code = NotFound desc = could not find container \"0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29\": container with ID starting with 0f39163300147a695e8b4c9291af6c0184a72f7a95ee83a527fdc278e7455c29 not found: ID does not exist" Sep 14 01:46:53.376893 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.376871 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:53.381286 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.381258 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e3d76dcd-openshift-default-8466dd7d8-j29sn"] Sep 14 01:46:53.832308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.832258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:53.928641 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:53.928600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:54.077132 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.077097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:54.178608 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.178525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:54.533258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.532479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:54.630581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.630547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:54.663691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.663657 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:46:54.664131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.664118 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:46:54.664131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.664132 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:46:54.664214 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.664151 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:46:54.664214 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.664157 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:46:54.664277 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.664222 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:46:54.666939 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.666921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.669093 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.669077 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b766dd7c-dockercfg-2sqtx\"" Sep 14 01:46:54.669154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.669121 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b766dd7c\"" Sep 14 01:46:54.676384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.676361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:46:54.788822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.788742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.788822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.788796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hps87\" (UniqueName: \"kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.890375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.890335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.890559 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.890387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hps87\" (UniqueName: \"kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.891034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.891013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.897828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.897806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hps87\" (UniqueName: \"kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87\") pod \"payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:54.978514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:54.978471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:55.139033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.138947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:55.218448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.218420 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a070084c-a411-4869-be23-86b3b9db6828" path="/var/lib/kubelet/pods/a070084c-a411-4869-be23-86b3b9db6828/volumes" Sep 14 01:46:55.253146 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.253122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:46:55.253763 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:46:55.253735 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b59f841_7c36_4fce_a49a_6a0b34294a25.slice/crio-42030043afd1505a667ed4272cae5031302cee14ccc753a1dd529b1c7ee25762 WatchSource:0}: Error finding container 42030043afd1505a667ed4272cae5031302cee14ccc753a1dd529b1c7ee25762: Status 404 returned error can't find the container with id 42030043afd1505a667ed4272cae5031302cee14ccc753a1dd529b1c7ee25762 Sep 14 01:46:55.369510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.369476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" event={"ID":"3b59f841-7c36-4fce-a49a-6a0b34294a25","Type":"ContainerStarted","Data":"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9"} Sep 14 01:46:55.369665 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.369535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" event={"ID":"3b59f841-7c36-4fce-a49a-6a0b34294a25","Type":"ContainerStarted","Data":"42030043afd1505a667ed4272cae5031302cee14ccc753a1dd529b1c7ee25762"} Sep 14 01:46:55.369665 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.369570 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:46:55.387031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.386978 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" podStartSLOduration=1.386964941 podStartE2EDuration="1.386964941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:55.384539247 +0000 UTC m=+960.617904529" watchObservedRunningTime="2026-09-14 01:46:55.386964941 +0000 UTC m=+960.620330199" Sep 14 01:46:55.874824 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:55.874784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:58.387801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.387722 2579 generic.go:356] "Generic (PLEG): container finished" podID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerID="8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a" exitCode=0 Sep 14 01:46:58.387801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.387794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerDied","Data":"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a"} Sep 14 01:46:58.388312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.388168 2579 scope.go:117] "RemoveContainer" containerID="8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a" Sep 14 01:46:58.389266 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.389241 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:58.389331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.389314 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:58.389408 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:58.389346 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:46:59.116598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:59.116555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:46:59.395294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:59.395214 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerStarted","Data":"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8"} Sep 14 01:46:59.656040 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:59.655953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:46:59.656247 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:59.656228 2579 patch_prober.go:28] interesting pod/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 14 01:46:59.656299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:46:59.656269 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 14 01:47:00.644494 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:00.644455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:00.655640 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:00.655610 2579 patch_prober.go:28] interesting pod/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 14 01:47:00.655773 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:00.655665 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 14 01:47:01.655653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:01.655614 2579 patch_prober.go:28] interesting pod/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 14 01:47:01.656160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:01.655681 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 14 01:47:02.655597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:02.655559 2579 patch_prober.go:28] interesting pod/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 14 01:47:02.655796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:02.655638 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 14 01:47:03.655901 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:03.655865 2579 patch_prober.go:28] interesting pod/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 14 01:47:03.656314 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:03.655926 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 14 01:47:04.662639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:04.662606 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:47:04.663123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:04.662990 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:47:04.663717 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:04.663697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:47:05.304349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:05.304318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:06.377470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:06.377435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:47:14.897497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:14.897398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:15.496815 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.496782 2579 generic.go:356] "Generic (PLEG): container finished" podID="6f5acee9-f465-487c-828f-78d9a4d78917" containerID="3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b" exitCode=0 Sep 14 01:47:15.497015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.496859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerDied","Data":"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b"} Sep 14 01:47:15.497356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.497339 2579 scope.go:117] "RemoveContainer" containerID="3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b" Sep 14 01:47:15.498542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.498517 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:15.498596 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.498583 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:15.498643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.498613 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:15.868563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.868486 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs"] Sep 14 01:47:15.869020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.869007 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a070084c-a411-4869-be23-86b3b9db6828" containerName="istio-proxy" Sep 14 01:47:15.871196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.871175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:15.873538 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.873385 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:47:15.873538 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.873415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:47:15.878326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.878268 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs"] Sep 14 01:47:15.928073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:15.928033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs\" (UID: \"a83f454a-ee19-4d86-beec-5d242407ec8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:16.028857 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.028801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs\" (UID: \"a83f454a-ee19-4d86-beec-5d242407ec8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:16.030009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.029976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs\" (UID: \"a83f454a-ee19-4d86-beec-5d242407ec8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:16.183737 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.183654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:16.246152 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.246112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:16.475463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.475433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs"] Sep 14 01:47:16.477394 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:47:16.477365 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83f454a_ee19_4d86_beec_5d242407ec8e.slice/crio-982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8 WatchSource:0}: Error finding container 982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8: Status 404 returned error can't find the container with id 982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8 Sep 14 01:47:16.504350 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.504319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" event={"ID":"a83f454a-ee19-4d86-beec-5d242407ec8e","Type":"ContainerStarted","Data":"982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8"} Sep 14 01:47:16.514444 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.506361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerStarted","Data":"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0"} Sep 14 01:47:16.896605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.896574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:16.915850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.915509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:16.917183 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.917148 2579 patch_prober.go:28] interesting pod/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:47:16.917300 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:16.917211 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 14 01:47:17.519210 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:17.519130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" event={"ID":"a83f454a-ee19-4d86-beec-5d242407ec8e","Type":"ContainerStarted","Data":"9b7ec24f4a0de71419df6c3928ca2e9280da9f86622855753231754435856bbd"} Sep 14 01:47:17.539793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:17.539746 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" podStartSLOduration=2.5397298470000003 podStartE2EDuration="2.539729847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:17.537473515 +0000 UTC m=+982.770838797" watchObservedRunningTime="2026-09-14 01:47:17.539729847 +0000 UTC m=+982.773095151" Sep 14 01:47:17.568634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:17.568600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:17.915144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:17.915005 2579 patch_prober.go:28] interesting pod/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:47:17.915380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:17.915349 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 14 01:47:18.204271 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.204169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:18.529830 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.529786 2579 generic.go:356] "Generic (PLEG): container finished" podID="a83f454a-ee19-4d86-beec-5d242407ec8e" containerID="9b7ec24f4a0de71419df6c3928ca2e9280da9f86622855753231754435856bbd" exitCode=0 Sep 14 01:47:18.530330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.529915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" event={"ID":"a83f454a-ee19-4d86-beec-5d242407ec8e","Type":"ContainerDied","Data":"9b7ec24f4a0de71419df6c3928ca2e9280da9f86622855753231754435856bbd"} Sep 14 01:47:18.863602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.863504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:18.915509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.915467 2579 patch_prober.go:28] interesting pod/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:47:18.915714 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:18.915558 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 14 01:47:19.463276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.463231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:19.690881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.690860 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:19.769445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.769409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca\") pod \"a83f454a-ee19-4d86-beec-5d242407ec8e\" (UID: \"a83f454a-ee19-4d86-beec-5d242407ec8e\") " Sep 14 01:47:19.769752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.769730 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a83f454a-ee19-4d86-beec-5d242407ec8e" (UID: "a83f454a-ee19-4d86-beec-5d242407ec8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:19.871144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.871107 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83f454a-ee19-4d86-beec-5d242407ec8e-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:19.915224 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.915179 2579 patch_prober.go:28] interesting pod/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:47:19.915406 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:19.915249 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 14 01:47:20.254298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.254258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:20.545367 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.545295 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" Sep 14 01:47:20.545509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.545293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs" event={"ID":"a83f454a-ee19-4d86-beec-5d242407ec8e","Type":"ContainerDied","Data":"982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8"} Sep 14 01:47:20.545509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.545412 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982e079477815301057980545009d8d80da93213c88a171eb63d8c72019fbdd8" Sep 14 01:47:20.915613 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.915515 2579 patch_prober.go:28] interesting pod/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:47:20.916187 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:20.916152 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 14 01:47:21.338474 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.338434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:21.736582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.736544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:21.763904 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.763873 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:47:21.764157 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.764130 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" podUID="3b59f841-7c36-4fce-a49a-6a0b34294a25" containerName="payload-processing" containerID="cri-o://e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9" gracePeriod=30 Sep 14 01:47:21.839542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.839493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:21.920457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.920426 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:21.922472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.922446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:21.924767 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.924745 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs"] Sep 14 01:47:21.926077 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.925982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:21.927297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.927273 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b766dd7c-gzkvs"] Sep 14 01:47:21.988697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:21.988610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:22.034594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.034568 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:47:22.053655 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.053616 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:47:22.076533 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.076448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:22.096210 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.096183 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume\") pod \"3b59f841-7c36-4fce-a49a-6a0b34294a25\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " Sep 14 01:47:22.096331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.096252 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hps87\" (UniqueName: \"kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87\") pod \"3b59f841-7c36-4fce-a49a-6a0b34294a25\" (UID: \"3b59f841-7c36-4fce-a49a-6a0b34294a25\") " Sep 14 01:47:22.096561 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.096533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b59f841-7c36-4fce-a49a-6a0b34294a25" (UID: "3b59f841-7c36-4fce-a49a-6a0b34294a25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:22.098199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.098174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87" (OuterVolumeSpecName: "kube-api-access-hps87") pod "3b59f841-7c36-4fce-a49a-6a0b34294a25" (UID: "3b59f841-7c36-4fce-a49a-6a0b34294a25"). InnerVolumeSpecName "kube-api-access-hps87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:22.197614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.197580 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b59f841-7c36-4fce-a49a-6a0b34294a25-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:22.197614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.197609 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hps87\" (UniqueName: \"kubernetes.io/projected/3b59f841-7c36-4fce-a49a-6a0b34294a25-kube-api-access-hps87\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:22.341324 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.341217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:22.542781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.542746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:22.557842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.557810 2579 generic.go:356] "Generic (PLEG): container finished" podID="3b59f841-7c36-4fce-a49a-6a0b34294a25" containerID="e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9" exitCode=0 Sep 14 01:47:22.557995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.557853 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" Sep 14 01:47:22.557995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.557853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" event={"ID":"3b59f841-7c36-4fce-a49a-6a0b34294a25","Type":"ContainerDied","Data":"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9"} Sep 14 01:47:22.557995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.557929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88" event={"ID":"3b59f841-7c36-4fce-a49a-6a0b34294a25","Type":"ContainerDied","Data":"42030043afd1505a667ed4272cae5031302cee14ccc753a1dd529b1c7ee25762"} Sep 14 01:47:22.557995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.557964 2579 scope.go:117] "RemoveContainer" containerID="e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9" Sep 14 01:47:22.558306 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.558286 2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" secret="" err="secret \"e2e-migrate-b766dd7c-gw-openshift-default-dockercfg-64q9v\" not found" Sep 14 01:47:22.568071 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.568047 2579 scope.go:117] "RemoveContainer" containerID="e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9" Sep 14 01:47:22.568315 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:47:22.568298 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9\": container with ID starting with e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9 not found: ID does not exist" containerID="e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9" Sep 14 01:47:22.568365 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.568323 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9"} err="failed to get container status \"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9\": rpc error: code = NotFound desc = could not find container \"e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9\": container with ID starting with e0a6580e58c344deea529db4111e45cea5d0be5a0129ebbf4bf98a8216287bb9 not found: ID does not exist" Sep 14 01:47:22.580175 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.580142 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:47:22.584181 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.584160 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b766dd7c-855b6d68bb-7jx88"] Sep 14 01:47:22.837986 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.837950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:47:22.838405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838392 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83f454a-ee19-4d86-beec-5d242407ec8e" containerName="revoke-keys" Sep 14 01:47:22.838461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838406 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83f454a-ee19-4d86-beec-5d242407ec8e" containerName="revoke-keys" Sep 14 01:47:22.838461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838428 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b59f841-7c36-4fce-a49a-6a0b34294a25" containerName="payload-processing" Sep 14 01:47:22.838461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838434 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b59f841-7c36-4fce-a49a-6a0b34294a25" containerName="payload-processing" Sep 14 01:47:22.838552 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838500 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83f454a-ee19-4d86-beec-5d242407ec8e" containerName="revoke-keys" Sep 14 01:47:22.838552 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.838514 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b59f841-7c36-4fce-a49a-6a0b34294a25" containerName="payload-processing" Sep 14 01:47:22.840849 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.840834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.843263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.843242 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-91e5cec3-openshift-default-dockercfg-zlwwg\"" Sep 14 01:47:22.851649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.851629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:47:22.858797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.858766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:22.905855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.905822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.905900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.905947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gkn7\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906252 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906252 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:22.906327 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:22.906250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.007637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gkn7\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.007957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008659 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008659 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.008659 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.008566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.010262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.010227 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.010491 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.010471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.015212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.015192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.015375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.015354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gkn7\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7\") pod \"e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.097189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.097106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:23.152923 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.152874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:23.216117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.215297 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b59f841-7c36-4fce-a49a-6a0b34294a25" path="/var/lib/kubelet/pods/3b59f841-7c36-4fce-a49a-6a0b34294a25/volumes" Sep 14 01:47:23.216117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.215813 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83f454a-ee19-4d86-beec-5d242407ec8e" path="/var/lib/kubelet/pods/a83f454a-ee19-4d86-beec-5d242407ec8e/volumes" Sep 14 01:47:23.318721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.318011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:23.433484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.433456 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:47:23.438217 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:47:23.438183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e086fa9_5d38_4abd_b71f_636e568a2c90.slice/crio-2ba4dc6f5f23dedbeacc2dd37b18a75bfaaed91192dd26a14f38c81e8f9de76a WatchSource:0}: Error finding container 2ba4dc6f5f23dedbeacc2dd37b18a75bfaaed91192dd26a14f38c81e8f9de76a: Status 404 returned error can't find the container with id 2ba4dc6f5f23dedbeacc2dd37b18a75bfaaed91192dd26a14f38c81e8f9de76a Sep 14 01:47:23.440434 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.440396 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:23.440525 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.440480 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:23.440566 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.440521 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:23.565651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.565613 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerStarted","Data":"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96"} Sep 14 01:47:23.565809 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.565681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerStarted","Data":"2ba4dc6f5f23dedbeacc2dd37b18a75bfaaed91192dd26a14f38c81e8f9de76a"} Sep 14 01:47:23.566899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.566841 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" containerID="cri-o://6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0" gracePeriod=30 Sep 14 01:47:23.584796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.584749 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podStartSLOduration=1.5847324 podStartE2EDuration="1.5847324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:23.583785062 +0000 UTC m=+988.817150345" watchObservedRunningTime="2026-09-14 01:47:23.5847324 +0000 UTC m=+988.818097658" Sep 14 01:47:23.876439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:23.876397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:24.153194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:24.153107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:24.158828 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:24.158803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:24.569934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:24.569886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:24.573016 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:24.572996 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:24.573836 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:24.573819 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:25.007746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:25.007717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:25.106882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:25.106847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:25.567035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:25.567000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:26.231823 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:26.231788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.038377 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.038345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.090315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.090280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.313520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.313430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.575856 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.575779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.664458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.664422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.720423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.720386 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:27.723189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.723169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.725746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.725726 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-91e5cec3\"" Sep 14 01:47:27.725846 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.725807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-91e5cec3-dockercfg-9tsqr\"" Sep 14 01:47:27.731504 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.731468 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:27.858443 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.858352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69hr\" (UniqueName: \"kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.858443 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.858408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.944975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.944933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:27.959964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.959936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69hr\" (UniqueName: \"kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.960134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.959977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.960666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.960648 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:27.969018 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:27.968989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69hr\" (UniqueName: \"kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr\") pod \"payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:28.035237 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.035208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:28.303457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.303431 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:28.305715 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:47:28.305670 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676e0f2e_2282_4c7a_b638_270f9bed3b19.slice/crio-06e39f9abc1f31aca661edcc59f73b4972af824ef893eaa1edff5736f9d722de WatchSource:0}: Error finding container 06e39f9abc1f31aca661edcc59f73b4972af824ef893eaa1edff5736f9d722de: Status 404 returned error can't find the container with id 06e39f9abc1f31aca661edcc59f73b4972af824ef893eaa1edff5736f9d722de Sep 14 01:47:28.605946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.605910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" event={"ID":"676e0f2e-2282-4c7a-b638-270f9bed3b19","Type":"ContainerStarted","Data":"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753"} Sep 14 01:47:28.606119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.606006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" event={"ID":"676e0f2e-2282-4c7a-b638-270f9bed3b19","Type":"ContainerStarted","Data":"06e39f9abc1f31aca661edcc59f73b4972af824ef893eaa1edff5736f9d722de"} Sep 14 01:47:28.606119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.606040 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:28.622694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.622639 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" podStartSLOduration=1.6226203209999999 podStartE2EDuration="1.622620321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:28.619956384 +0000 UTC m=+993.853321661" watchObservedRunningTime="2026-09-14 01:47:28.622620321 +0000 UTC m=+993.855985580" Sep 14 01:47:28.633720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.633686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:28.825324 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.825298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:28.970457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970366 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970655 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970684 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970786 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970812 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgwr9\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9\") pod \"6f5acee9-f465-487c-828f-78d9a4d78917\" (UID: \"6f5acee9-f465-487c-828f-78d9a4d78917\") " Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970850 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:47:28.970920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.970877 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:28.971284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.971219 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:28.971284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.971239 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:28.971284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.971254 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:28.971284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.971267 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f5acee9-f465-487c-828f-78d9a4d78917-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:28.971284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.971279 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:28.973193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.973168 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:47:28.973331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.973312 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:28.973381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.973341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9" (OuterVolumeSpecName: "kube-api-access-jgwr9") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "kube-api-access-jgwr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:28.973381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:28.973345 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f5acee9-f465-487c-828f-78d9a4d78917" (UID: "6f5acee9-f465-487c-828f-78d9a4d78917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:47:29.072122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.072099 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f5acee9-f465-487c-828f-78d9a4d78917-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:29.072122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.072120 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwr9\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-kube-api-access-jgwr9\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:29.072268 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.072132 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f5acee9-f465-487c-828f-78d9a4d78917-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:29.072268 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.072142 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f5acee9-f465-487c-828f-78d9a4d78917-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:29.358127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.358094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:29.617578 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.617543 2579 generic.go:356] "Generic (PLEG): container finished" podID="6f5acee9-f465-487c-828f-78d9a4d78917" containerID="6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0" exitCode=0 Sep 14 01:47:29.617782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.617602 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" Sep 14 01:47:29.617782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.617608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerDied","Data":"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0"} Sep 14 01:47:29.617782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.617661 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z" event={"ID":"6f5acee9-f465-487c-828f-78d9a4d78917","Type":"ContainerDied","Data":"282f8bfc6a1b520d406d2b2afaca1da19a99931fe94cb27d6b119bb0c13f8495"} Sep 14 01:47:29.617782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.617683 2579 scope.go:117] "RemoveContainer" containerID="6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0" Sep 14 01:47:29.626944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.626922 2579 scope.go:117] "RemoveContainer" containerID="3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b" Sep 14 01:47:29.635850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.635833 2579 scope.go:117] "RemoveContainer" containerID="6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0" Sep 14 01:47:29.636121 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:47:29.636099 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0\": container with ID starting with 6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0 not found: ID does not exist" containerID="6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0" Sep 14 01:47:29.636188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.636133 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0"} err="failed to get container status \"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0\": rpc error: code = NotFound desc = could not find container \"6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0\": container with ID starting with 6eb7dcb721f80ce930bd102921d00e530f85508f57c199c658613c0deae0a5d0 not found: ID does not exist" Sep 14 01:47:29.636188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.636156 2579 scope.go:117] "RemoveContainer" containerID="3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b" Sep 14 01:47:29.636398 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:47:29.636381 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b\": container with ID starting with 3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b not found: ID does not exist" containerID="3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b" Sep 14 01:47:29.636438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.636406 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b"} err="failed to get container status \"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b\": rpc error: code = NotFound desc = could not find container \"3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b\": container with ID starting with 3d6bb106f647d1900ec288704bd0b7f6708aae27d24ef7a0f1052a057fa97c2b not found: ID does not exist" Sep 14 01:47:29.640915 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.640894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:47:29.644194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.644176 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b766dd7c-gw-openshift-default-66dfd57bdd-82m2z"] Sep 14 01:47:29.962692 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:29.962618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:31.210817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:31.210787 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" path="/var/lib/kubelet/pods/6f5acee9-f465-487c-828f-78d9a4d78917/volumes" Sep 14 01:47:34.425772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:34.425731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:39.620165 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:39.620137 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:49.004270 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.004229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:49.613439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.613399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:49.668295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.668263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:49.717491 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.717460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:49.734503 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.734478 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerID="181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96" exitCode=0 Sep 14 01:47:49.734634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.734552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerDied","Data":"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96"} Sep 14 01:47:49.735019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.735002 2579 scope.go:117] "RemoveContainer" containerID="181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96" Sep 14 01:47:49.736320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.736294 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:49.736389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.736365 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:49.736424 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:49.736395 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:50.438374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.438340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:50.688475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688401 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv"] Sep 14 01:47:50.688894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688881 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.688941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688897 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.688941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688910 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.688941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688916 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.689031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.688995 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.689031 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.689005 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5acee9-f465-487c-828f-78d9a4d78917" containerName="istio-proxy" Sep 14 01:47:50.692265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.692250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:50.695643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.695619 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:47:50.695643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.695639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:47:50.699267 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.699247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv"] Sep 14 01:47:50.744854 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.744824 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerStarted","Data":"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38"} Sep 14 01:47:50.795039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.795001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv\" (UID: \"9eadca64-6588-4afc-a7a0-49faf0c53651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:50.896049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.896000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv\" (UID: \"9eadca64-6588-4afc-a7a0-49faf0c53651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:50.896661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:50.896642 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv\" (UID: \"9eadca64-6588-4afc-a7a0-49faf0c53651\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:51.004852 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.004814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:51.153392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.153355 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:51.154289 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.154265 2579 patch_prober.go:28] interesting pod/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 14 01:47:51.154357 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.154319 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 14 01:47:51.163596 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.163512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:51.301644 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.301617 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv"] Sep 14 01:47:51.303478 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:47:51.303446 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eadca64_6588_4afc_a7a0_49faf0c53651.slice/crio-275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5 WatchSource:0}: Error finding container 275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5: Status 404 returned error can't find the container with id 275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5 Sep 14 01:47:51.755658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.755599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" event={"ID":"9eadca64-6588-4afc-a7a0-49faf0c53651","Type":"ContainerStarted","Data":"9827016ce2f98a9625ddfec0cf0674d21b6952c3ce12fe493585ec2833942617"} Sep 14 01:47:51.756100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.755707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" event={"ID":"9eadca64-6588-4afc-a7a0-49faf0c53651","Type":"ContainerStarted","Data":"275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5"} Sep 14 01:47:51.771402 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:51.771359 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" podStartSLOduration=1.7713444360000001 podStartE2EDuration="1.771344436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:51.769219862 +0000 UTC m=+1017.002585120" watchObservedRunningTime="2026-09-14 01:47:51.771344436 +0000 UTC m=+1017.004709693" Sep 14 01:47:52.046500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.046459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:52.154293 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.154254 2579 patch_prober.go:28] interesting pod/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 14 01:47:52.154471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.154316 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 14 01:47:52.646624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.646523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:52.762880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.762843 2579 generic.go:356] "Generic (PLEG): container finished" podID="9eadca64-6588-4afc-a7a0-49faf0c53651" containerID="9827016ce2f98a9625ddfec0cf0674d21b6952c3ce12fe493585ec2833942617" exitCode=0 Sep 14 01:47:52.763283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:52.762934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" event={"ID":"9eadca64-6588-4afc-a7a0-49faf0c53651","Type":"ContainerDied","Data":"9827016ce2f98a9625ddfec0cf0674d21b6952c3ce12fe493585ec2833942617"} Sep 14 01:47:53.153837 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:53.153798 2579 patch_prober.go:28] interesting pod/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 14 01:47:53.154002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:53.153857 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 14 01:47:53.537752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:53.537719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:53.941020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:53.940993 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:54.034894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.034867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca\") pod \"9eadca64-6588-4afc-a7a0-49faf0c53651\" (UID: \"9eadca64-6588-4afc-a7a0-49faf0c53651\") " Sep 14 01:47:54.035293 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.035272 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eadca64-6588-4afc-a7a0-49faf0c53651" (UID: "9eadca64-6588-4afc-a7a0-49faf0c53651"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:54.136746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.136661 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eadca64-6588-4afc-a7a0-49faf0c53651-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:54.153678 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.153648 2579 patch_prober.go:28] interesting pod/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 14 01:47:54.153802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.153700 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 14 01:47:54.245173 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.245142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:54.573676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.573641 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:54.776044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.776015 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" Sep 14 01:47:54.776044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.776020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv" event={"ID":"9eadca64-6588-4afc-a7a0-49faf0c53651","Type":"ContainerDied","Data":"275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5"} Sep 14 01:47:54.776269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.776089 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275890b358f98667a470155ab1cfa97893245c79fc96c188dcc0527e7c23dbf5" Sep 14 01:47:54.840287 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:54.840198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:55.153343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:55.153263 2579 patch_prober.go:28] interesting pod/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 14 01:47:55.153343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:55.153312 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 14 01:47:55.643143 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:55.643105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:56.157448 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:56.157417 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:56.158292 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:56.158274 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:47:56.450757 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:56.450668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.081328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.081285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.102856 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.102822 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:57.103298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.103266 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" podUID="676e0f2e-2282-4c7a-b638-270f9bed3b19" containerName="payload-processing" containerID="cri-o://ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753" gracePeriod=30 Sep 14 01:47:57.174930 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.174903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.245450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.245421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.245605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.245565 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv"] Sep 14 01:47:57.249222 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.249196 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-91e5cec3-rlzkv"] Sep 14 01:47:57.296340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.296315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.391984 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.391957 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:57.472585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.472550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume\") pod \"676e0f2e-2282-4c7a-b638-270f9bed3b19\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " Sep 14 01:47:57.472777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.472643 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m69hr\" (UniqueName: \"kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr\") pod \"676e0f2e-2282-4c7a-b638-270f9bed3b19\" (UID: \"676e0f2e-2282-4c7a-b638-270f9bed3b19\") " Sep 14 01:47:57.473025 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.472998 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "676e0f2e-2282-4c7a-b638-270f9bed3b19" (UID: "676e0f2e-2282-4c7a-b638-270f9bed3b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:57.474788 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.474768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr" (OuterVolumeSpecName: "kube-api-access-m69hr") pod "676e0f2e-2282-4c7a-b638-270f9bed3b19" (UID: "676e0f2e-2282-4c7a-b638-270f9bed3b19"). InnerVolumeSpecName "kube-api-access-m69hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:57.573729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.573699 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e0f2e-2282-4c7a-b638-270f9bed3b19-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:57.573729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.573724 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69hr\" (UniqueName: \"kubernetes.io/projected/676e0f2e-2282-4c7a-b638-270f9bed3b19-kube-api-access-m69hr\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:47:57.725598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.725509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:57.800282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.800247 2579 generic.go:356] "Generic (PLEG): container finished" podID="676e0f2e-2282-4c7a-b638-270f9bed3b19" containerID="ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753" exitCode=0 Sep 14 01:47:57.800455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.800298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" Sep 14 01:47:57.800455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.800331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" event={"ID":"676e0f2e-2282-4c7a-b638-270f9bed3b19","Type":"ContainerDied","Data":"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753"} Sep 14 01:47:57.800455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.800384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c" event={"ID":"676e0f2e-2282-4c7a-b638-270f9bed3b19","Type":"ContainerDied","Data":"06e39f9abc1f31aca661edcc59f73b4972af824ef893eaa1edff5736f9d722de"} Sep 14 01:47:57.800455 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.800408 2579 scope.go:117] "RemoveContainer" containerID="ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753" Sep 14 01:47:57.810899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.810879 2579 scope.go:117] "RemoveContainer" containerID="ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753" Sep 14 01:47:57.811225 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:47:57.811205 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753\": container with ID starting with ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753 not found: ID does not exist" containerID="ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753" Sep 14 01:47:57.811307 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.811237 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753"} err="failed to get container status \"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753\": rpc error: code = NotFound desc = could not find container \"ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753\": container with ID starting with ea33399b43c0293398773204f8f268d22f48cfb3fa2d8313d793dd5bb60e9753 not found: ID does not exist" Sep 14 01:47:57.823953 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.823930 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:57.826539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:57.826519 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-91e5cec3-7557856b7f-n4m7c"] Sep 14 01:47:58.343357 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:58.343324 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:47:58.343840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:58.343527 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" containerID="cri-o://97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38" gracePeriod=30 Sep 14 01:47:58.354603 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:58.354543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:58.455891 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:58.455789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:58.779227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:58.779190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:59.063396 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.063309 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:47:59.063898 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.063877 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eadca64-6588-4afc-a7a0-49faf0c53651" containerName="revoke-keys" Sep 14 01:47:59.064009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.063900 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eadca64-6588-4afc-a7a0-49faf0c53651" containerName="revoke-keys" Sep 14 01:47:59.064009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.063920 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="676e0f2e-2282-4c7a-b638-270f9bed3b19" containerName="payload-processing" Sep 14 01:47:59.064009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.063928 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="676e0f2e-2282-4c7a-b638-270f9bed3b19" containerName="payload-processing" Sep 14 01:47:59.064213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.064078 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="676e0f2e-2282-4c7a-b638-270f9bed3b19" containerName="payload-processing" Sep 14 01:47:59.064213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.064096 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eadca64-6588-4afc-a7a0-49faf0c53651" containerName="revoke-keys" Sep 14 01:47:59.068509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.068491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.071743 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.071727 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b351ffdb-openshift-default-dockercfg-xmf6c\"" Sep 14 01:47:59.091740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.091697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:47:59.111447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.111411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9lr\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.191946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.192031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.193110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.192071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.210389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.210358 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="676e0f2e-2282-4c7a-b638-270f9bed3b19" path="/var/lib/kubelet/pods/676e0f2e-2282-4c7a-b638-270f9bed3b19/volumes" Sep 14 01:47:59.210705 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.210692 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eadca64-6588-4afc-a7a0-49faf0c53651" path="/var/lib/kubelet/pods/9eadca64-6588-4afc-a7a0-49faf0c53651/volumes" Sep 14 01:47:59.229973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.229943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:59.292639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.292831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.292831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.292831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.292900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9lr\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293230 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293390 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.293612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.293592 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.295380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.295352 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.295598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.295581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.303095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.303074 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.303265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.303247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9lr\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr\") pod \"e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.378574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.378488 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:47:59.485913 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.485868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:47:59.655965 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.655940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:47:59.657577 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:47:59.657550 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57041dc_ae03_4a50_a21f_2d47ad840690.slice/crio-900afb0835b0a53894d24fafff5bb5a80b4cfdf7b5be775543c22b5013a1d8a4 WatchSource:0}: Error finding container 900afb0835b0a53894d24fafff5bb5a80b4cfdf7b5be775543c22b5013a1d8a4: Status 404 returned error can't find the container with id 900afb0835b0a53894d24fafff5bb5a80b4cfdf7b5be775543c22b5013a1d8a4 Sep 14 01:47:59.660120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.660083 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:59.660212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.660166 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:59.660251 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.660213 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:47:59.817396 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.817361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerStarted","Data":"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909"} Sep 14 01:47:59.817578 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.817418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerStarted","Data":"900afb0835b0a53894d24fafff5bb5a80b4cfdf7b5be775543c22b5013a1d8a4"} Sep 14 01:47:59.834906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.834799 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podStartSLOduration=0.834776429 podStartE2EDuration="834.776429ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:59.834170596 +0000 UTC m=+1025.067535892" watchObservedRunningTime="2026-09-14 01:47:59.834776429 +0000 UTC m=+1025.068141687" Sep 14 01:47:59.871781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:47:59.871746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:00.378884 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:00.378852 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:00.383948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:00.383923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:00.659194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:00.659106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:00.823559 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:00.823528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:00.824634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:00.824614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:01.267288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:01.267256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:01.367052 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:01.367018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:01.563194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:01.563102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:02.208196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:02.208159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:03.032732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.032695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:03.595308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.595283 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:48:03.718328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.718234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:03.741513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741580 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741605 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741634 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741718 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gkn7\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo\") pod \"0e086fa9-5d38-4abd-b71f-636e568a2c90\" (UID: \"0e086fa9-5d38-4abd-b71f-636e568a2c90\") " Sep 14 01:48:03.741998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:03.741998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.741868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742095 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742111 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742195 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742213 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742226 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:03.742284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.742240 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.743939 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.743912 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:03.744094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.744044 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7" (OuterVolumeSpecName: "kube-api-access-7gkn7") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "kube-api-access-7gkn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:03.744219 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.744185 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:03.744297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.744278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e086fa9-5d38-4abd-b71f-636e568a2c90" (UID: "0e086fa9-5d38-4abd-b71f-636e568a2c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:48:03.841435 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.841401 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerID="97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38" exitCode=0 Sep 14 01:48:03.841584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.841447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" Sep 14 01:48:03.841584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.841478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerDied","Data":"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38"} Sep 14 01:48:03.841584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.841534 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8" event={"ID":"0e086fa9-5d38-4abd-b71f-636e568a2c90","Type":"ContainerDied","Data":"2ba4dc6f5f23dedbeacc2dd37b18a75bfaaed91192dd26a14f38c81e8f9de76a"} Sep 14 01:48:03.841584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.841557 2579 scope.go:117] "RemoveContainer" containerID="97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38" Sep 14 01:48:03.842736 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.842719 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.842798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.842738 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.842798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.842747 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e086fa9-5d38-4abd-b71f-636e568a2c90-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.842798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.842755 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e086fa9-5d38-4abd-b71f-636e568a2c90-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.842798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.842762 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gkn7\" (UniqueName: \"kubernetes.io/projected/0e086fa9-5d38-4abd-b71f-636e568a2c90-kube-api-access-7gkn7\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:03.851152 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.851136 2579 scope.go:117] "RemoveContainer" containerID="181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96" Sep 14 01:48:03.859189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.859171 2579 scope.go:117] "RemoveContainer" containerID="97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38" Sep 14 01:48:03.859440 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:48:03.859423 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38\": container with ID starting with 97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38 not found: ID does not exist" containerID="97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38" Sep 14 01:48:03.859499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.859448 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38"} err="failed to get container status \"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38\": rpc error: code = NotFound desc = could not find container \"97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38\": container with ID starting with 97ddb359174428f1212921ee4a822860ee80e620d13bf7b11f08e44629064d38 not found: ID does not exist" Sep 14 01:48:03.859499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.859466 2579 scope.go:117] "RemoveContainer" containerID="181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96" Sep 14 01:48:03.859743 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:48:03.859724 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96\": container with ID starting with 181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96 not found: ID does not exist" containerID="181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96" Sep 14 01:48:03.859792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.859748 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96"} err="failed to get container status \"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96\": rpc error: code = NotFound desc = could not find container \"181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96\": container with ID starting with 181f45af2d4395bc8c7eb919cca51c7ff3ed979aebd41acdea6e49137db3bd96 not found: ID does not exist" Sep 14 01:48:03.865625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.865602 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:48:03.868580 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:03.868559 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-91e5cec3-openshift-default-744fbfc79b-rhmq8"] Sep 14 01:48:04.355882 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:04.355848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.149928 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.149890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.209312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.209274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.212950 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.212918 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" path="/var/lib/kubelet/pods/0e086fa9-5d38-4abd-b71f-636e568a2c90/volumes" Sep 14 01:48:05.348284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.348252 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm"] Sep 14 01:48:05.348708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.348695 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.348753 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.348709 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.348796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.348786 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.348829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.348798 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.353905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.353879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:05.356289 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.356269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:48:05.356661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.356324 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:48:05.358629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.358607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm"] Sep 14 01:48:05.390974 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.390930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.458438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.458362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bc2e37-brfqm\" (UID: \"5d1b4809-5293-4758-8d06-11b70e7a5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:05.560073 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.560039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bc2e37-brfqm\" (UID: \"5d1b4809-5293-4758-8d06-11b70e7a5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:05.560703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.560685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bc2e37-brfqm\" (UID: \"5d1b4809-5293-4758-8d06-11b70e7a5d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:05.667018 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.666985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:05.702820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.702784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.799517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.799485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:05.848850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.848816 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:05.849490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.849469 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.849539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.849495 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e086fa9-5d38-4abd-b71f-636e568a2c90" containerName="istio-proxy" Sep 14 01:48:05.854238 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.854215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:05.856670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.856642 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b351ffdb\"" Sep 14 01:48:05.856777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.856740 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b351ffdb-dockercfg-2dtlk\"" Sep 14 01:48:05.860791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.860768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:05.956091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.952324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm"] Sep 14 01:48:05.967897 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.965451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9895c\" (UniqueName: \"kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:05.967897 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:05.965522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.066972 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.066942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9895c\" (UniqueName: \"kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.067144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.066991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.067732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.067712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.074389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.074362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9895c\" (UniqueName: \"kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c\") pod \"payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.129275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.129241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:06.166829 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.166801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.451400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.451361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:06.834105 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.834051 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:06.861991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.861953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" event={"ID":"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb","Type":"ContainerStarted","Data":"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae"} Sep 14 01:48:06.862167 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.862024 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" event={"ID":"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb","Type":"ContainerStarted","Data":"f12b6a9650062c97a27b5a056777d439e70123e5f111600587fbf93786e15c33"} Sep 14 01:48:06.862167 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.862074 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:06.863702 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.863675 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d1b4809-5293-4758-8d06-11b70e7a5d3c" containerID="2c7f84b95c6cd4be60c414443a5e425d6d2a7a3ff348968d3f77dd5ba91d30ac" exitCode=0 Sep 14 01:48:06.863816 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.863725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" event={"ID":"5d1b4809-5293-4758-8d06-11b70e7a5d3c","Type":"ContainerDied","Data":"2c7f84b95c6cd4be60c414443a5e425d6d2a7a3ff348968d3f77dd5ba91d30ac"} Sep 14 01:48:06.863816 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.863756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" event={"ID":"5d1b4809-5293-4758-8d06-11b70e7a5d3c","Type":"ContainerStarted","Data":"7a72158756327ba0d41df718627412f3657ce06549996bebea084b3c54e1f434"} Sep 14 01:48:06.878935 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:06.878887 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" podStartSLOduration=1.878873346 podStartE2EDuration="1.878873346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:48:06.87679016 +0000 UTC m=+1032.110155440" watchObservedRunningTime="2026-09-14 01:48:06.878873346 +0000 UTC m=+1032.112238603" Sep 14 01:48:07.630174 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:07.630137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:08.015671 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.015645 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:08.088542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.088509 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca\") pod \"5d1b4809-5293-4758-8d06-11b70e7a5d3c\" (UID: \"5d1b4809-5293-4758-8d06-11b70e7a5d3c\") " Sep 14 01:48:08.088913 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.088887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d1b4809-5293-4758-8d06-11b70e7a5d3c" (UID: "5d1b4809-5293-4758-8d06-11b70e7a5d3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:08.189899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.189809 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b4809-5293-4758-8d06-11b70e7a5d3c-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:08.261418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.261383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:08.876995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.876961 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" Sep 14 01:48:08.877419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.877014 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm" event={"ID":"5d1b4809-5293-4758-8d06-11b70e7a5d3c","Type":"ContainerDied","Data":"7a72158756327ba0d41df718627412f3657ce06549996bebea084b3c54e1f434"} Sep 14 01:48:08.877419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.877092 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a72158756327ba0d41df718627412f3657ce06549996bebea084b3c54e1f434" Sep 14 01:48:08.936870 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:08.936837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:09.549995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:09.549959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:10.249690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:10.249655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:13.107285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:13.107249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:15.491093 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:15.491035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.149840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.149788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.249634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.249579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.323254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.323220 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm"] Sep 14 01:48:16.325039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.325005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.325692 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.325672 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bc2e37-brfqm"] Sep 14 01:48:16.373464 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.373430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.702383 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.702348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:16.806192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:16.806155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:17.212760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:17.212718 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1b4809-5293-4758-8d06-11b70e7a5d3c" path="/var/lib/kubelet/pods/5d1b4809-5293-4758-8d06-11b70e7a5d3c/volumes" Sep 14 01:48:17.536508 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:17.536469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:17.871804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:17.871722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:18.279732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:18.279700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:19.031825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:19.031780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:19.735302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:19.735273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:20.446089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.446030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:20.908547 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.908508 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm"] Sep 14 01:48:20.909269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.909242 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1b4809-5293-4758-8d06-11b70e7a5d3c" containerName="revoke-keys" Sep 14 01:48:20.909269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.909267 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1b4809-5293-4758-8d06-11b70e7a5d3c" containerName="revoke-keys" Sep 14 01:48:20.909464 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.909390 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1b4809-5293-4758-8d06-11b70e7a5d3c" containerName="revoke-keys" Sep 14 01:48:20.912979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.912950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:20.916074 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.916034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:48:20.916553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.916534 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:48:20.917731 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.917634 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm"] Sep 14 01:48:20.942087 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:20.937096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm\" (UID: \"1772d450-ce8c-4980-a8c9-fed084bfec94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:21.038282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.038249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm\" (UID: \"1772d450-ce8c-4980-a8c9-fed084bfec94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:21.038998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.038979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm\" (UID: \"1772d450-ce8c-4980-a8c9-fed084bfec94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:21.233731 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.233658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:21.337566 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.337528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:21.511127 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.511032 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm"] Sep 14 01:48:21.727754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.727718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:21.966549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.966504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" event={"ID":"1772d450-ce8c-4980-a8c9-fed084bfec94","Type":"ContainerStarted","Data":"3fb3056aa37f2439459a6f88ef2031e1c3d229fd140ed381dce3db722bd2fc1f"} Sep 14 01:48:21.966727 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.966611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" event={"ID":"1772d450-ce8c-4980-a8c9-fed084bfec94","Type":"ContainerStarted","Data":"963a19475050d8f559ea941efc2ce79241294abd949ce7aa031e732a8382c4ee"} Sep 14 01:48:21.971459 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.971436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:21.982627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:21.982580 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" podStartSLOduration=1.9825663420000001 podStartE2EDuration="1.982566342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:48:21.981745508 +0000 UTC m=+1047.215110764" watchObservedRunningTime="2026-09-14 01:48:21.982566342 +0000 UTC m=+1047.215931599" Sep 14 01:48:22.646371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:22.646330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:22.974352 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:22.974314 2579 generic.go:356] "Generic (PLEG): container finished" podID="1772d450-ce8c-4980-a8c9-fed084bfec94" containerID="3fb3056aa37f2439459a6f88ef2031e1c3d229fd140ed381dce3db722bd2fc1f" exitCode=0 Sep 14 01:48:22.974512 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:22.974412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" event={"ID":"1772d450-ce8c-4980-a8c9-fed084bfec94","Type":"ContainerDied","Data":"3fb3056aa37f2439459a6f88ef2031e1c3d229fd140ed381dce3db722bd2fc1f"} Sep 14 01:48:23.261986 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:23.261949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:24.065657 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.065598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:24.128419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.127931 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:24.166955 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.166922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca\") pod \"1772d450-ce8c-4980-a8c9-fed084bfec94\" (UID: \"1772d450-ce8c-4980-a8c9-fed084bfec94\") " Sep 14 01:48:24.167308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.167283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1772d450-ce8c-4980-a8c9-fed084bfec94" (UID: "1772d450-ce8c-4980-a8c9-fed084bfec94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:24.268467 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.268369 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1772d450-ce8c-4980-a8c9-fed084bfec94-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:24.671623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.671535 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:24.986751 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.986719 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" Sep 14 01:48:24.986946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.986758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm" event={"ID":"1772d450-ce8c-4980-a8c9-fed084bfec94","Type":"ContainerDied","Data":"963a19475050d8f559ea941efc2ce79241294abd949ce7aa031e732a8382c4ee"} Sep 14 01:48:24.986946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:24.986803 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963a19475050d8f559ea941efc2ce79241294abd949ce7aa031e732a8382c4ee" Sep 14 01:48:25.581223 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.581186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:25.994241 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.994212 2579 generic.go:356] "Generic (PLEG): container finished" podID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerID="e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909" exitCode=0 Sep 14 01:48:25.994453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.994252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerDied","Data":"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909"} Sep 14 01:48:25.994875 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.994713 2579 scope.go:117] "RemoveContainer" containerID="e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909" Sep 14 01:48:25.996007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.995976 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:48:25.996096 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.996042 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:48:25.996142 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:25.996096 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:48:27.002195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:27.002162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerStarted","Data":"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e"} Sep 14 01:48:27.379513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:27.379415 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:27.380876 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:27.380850 2579 patch_prober.go:28] interesting pod/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 14 01:48:27.381008 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:27.380903 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 14 01:48:27.805717 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:27.805681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:28.379129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:28.379091 2579 patch_prober.go:28] interesting pod/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 14 01:48:28.379544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:28.379152 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 14 01:48:29.379287 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:29.379247 2579 patch_prober.go:28] interesting pod/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 14 01:48:29.379687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:29.379305 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 14 01:48:30.378923 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:30.378890 2579 patch_prober.go:28] interesting pod/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 14 01:48:30.379120 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:30.378943 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 14 01:48:30.823762 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:30.823727 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:31.379356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.379321 2579 patch_prober.go:28] interesting pod/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 14 01:48:31.379531 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.379377 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 14 01:48:31.453261 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.453190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:31.472661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.472633 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:31.473049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.473018 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" podUID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" containerName="payload-processing" containerID="cri-o://cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae" gracePeriod=30 Sep 14 01:48:31.551441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.551406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:31.636161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.632676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:31.636161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.635115 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm"] Sep 14 01:48:31.637411 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.637392 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b351ffdb-78hcm"] Sep 14 01:48:31.690115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.690052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:31.762499 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.762453 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:48:31.762991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.762948 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" containerID="cri-o://1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e" gracePeriod=30 Sep 14 01:48:31.788953 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.788768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:31.844821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.844573 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:31.968112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.968004 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume\") pod \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " Sep 14 01:48:31.968281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.968151 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9895c\" (UniqueName: \"kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c\") pod \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\" (UID: \"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb\") " Sep 14 01:48:31.968356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.968335 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" (UID: "ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:31.970512 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:31.970488 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c" (OuterVolumeSpecName: "kube-api-access-9895c") pod "ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" (UID: "ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb"). InnerVolumeSpecName "kube-api-access-9895c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:32.004104 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.004069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:32.044044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.044008 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" containerID="cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae" exitCode=0 Sep 14 01:48:32.044225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.044079 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" Sep 14 01:48:32.044225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.044089 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" event={"ID":"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb","Type":"ContainerDied","Data":"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae"} Sep 14 01:48:32.044225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.044165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db" event={"ID":"ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb","Type":"ContainerDied","Data":"f12b6a9650062c97a27b5a056777d439e70123e5f111600587fbf93786e15c33"} Sep 14 01:48:32.044225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.044188 2579 scope.go:117] "RemoveContainer" containerID="cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae" Sep 14 01:48:32.054304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.054287 2579 scope.go:117] "RemoveContainer" containerID="cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae" Sep 14 01:48:32.054568 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:48:32.054548 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae\": container with ID starting with cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae not found: ID does not exist" containerID="cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae" Sep 14 01:48:32.054632 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.054575 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae"} err="failed to get container status \"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae\": rpc error: code = NotFound desc = could not find container \"cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae\": container with ID starting with cef608125fe87029747da55770e0573680531199ed762174c9b7d2ad6a25f4ae not found: ID does not exist" Sep 14 01:48:32.072981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.067084 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:32.072981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.070439 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:32.072981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.070463 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9895c\" (UniqueName: \"kubernetes.io/projected/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb-kube-api-access-9895c\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:32.072981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.072892 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b351ffdb-c4dbb7c69-mh8db"] Sep 14 01:48:32.455447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.455410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:32.553347 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.553304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:32.608934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.608892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:32.763089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:32.763023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:33.210513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:33.210438 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1772d450-ce8c-4980-a8c9-fed084bfec94" path="/var/lib/kubelet/pods/1772d450-ce8c-4980-a8c9-fed084bfec94/volumes" Sep 14 01:48:33.210860 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:33.210806 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" path="/var/lib/kubelet/pods/ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb/volumes" Sep 14 01:48:33.518030 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:33.517994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:34.068663 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:34.068621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:34.241218 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:34.241168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:34.860268 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:34.860238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:37.736505 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.736480 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:37.822944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.822852 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j9lr\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.822944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.822902 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.822946 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823095 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823151 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823177 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823222 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823263 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs\") pod \"e57041dc-ae03-4a50-a21f-2d47ad840690\" (UID: \"e57041dc-ae03-4a50-a21f-2d47ad840690\") " Sep 14 01:48:37.823470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:37.823697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:37.823697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823661 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:37.823697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823675 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.823697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823696 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57041dc-ae03-4a50-a21f-2d47ad840690-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.823841 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.823720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data" (OuterVolumeSpecName: "istio-data") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:37.824089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.824043 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:37.825310 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.825282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:48:37.825691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.825667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:48:37.825777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.825687 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr" (OuterVolumeSpecName: "kube-api-access-8j9lr") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "kube-api-access-8j9lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:37.825777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.825706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token" (OuterVolumeSpecName: "istio-token") pod "e57041dc-ae03-4a50-a21f-2d47ad840690" (UID: "e57041dc-ae03-4a50-a21f-2d47ad840690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:37.924800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924773 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924796 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924806 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9lr\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-kube-api-access-8j9lr\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924816 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924824 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924832 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57041dc-ae03-4a50-a21f-2d47ad840690-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:37.924989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:37.924840 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57041dc-ae03-4a50-a21f-2d47ad840690-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:48:38.086699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.086612 2579 generic.go:356] "Generic (PLEG): container finished" podID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerID="1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e" exitCode=0 Sep 14 01:48:38.086699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.086668 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" Sep 14 01:48:38.086699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.086686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerDied","Data":"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e"} Sep 14 01:48:38.086946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.086739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp" event={"ID":"e57041dc-ae03-4a50-a21f-2d47ad840690","Type":"ContainerDied","Data":"900afb0835b0a53894d24fafff5bb5a80b4cfdf7b5be775543c22b5013a1d8a4"} Sep 14 01:48:38.086946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.086770 2579 scope.go:117] "RemoveContainer" containerID="1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e" Sep 14 01:48:38.096673 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.096655 2579 scope.go:117] "RemoveContainer" containerID="e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909" Sep 14 01:48:38.105002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.104985 2579 scope.go:117] "RemoveContainer" containerID="1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e" Sep 14 01:48:38.106689 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:48:38.106652 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e\": container with ID starting with 1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e not found: ID does not exist" containerID="1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e" Sep 14 01:48:38.106763 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.106701 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e"} err="failed to get container status \"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e\": rpc error: code = NotFound desc = could not find container \"1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e\": container with ID starting with 1593e173710dc31f675265fbbd8a37c4d819b6b4974943feb9cfd7460be52a7e not found: ID does not exist" Sep 14 01:48:38.106763 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.106725 2579 scope.go:117] "RemoveContainer" containerID="e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909" Sep 14 01:48:38.107051 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:48:38.107032 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909\": container with ID starting with e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909 not found: ID does not exist" containerID="e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909" Sep 14 01:48:38.107136 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.107069 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909"} err="failed to get container status \"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909\": rpc error: code = NotFound desc = could not find container \"e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909\": container with ID starting with e79baa7f860263b4f507dba1e9f28e87f47958f829735d5a5aa69bcd88884909 not found: ID does not exist" Sep 14 01:48:38.108324 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.108305 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:48:38.117497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:38.117474 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b351ffdb-openshift-default-68b8889856-jdwrp"] Sep 14 01:48:39.210359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:39.210319 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" path="/var/lib/kubelet/pods/e57041dc-ae03-4a50-a21f-2d47ad840690/volumes" Sep 14 01:48:39.702197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:39.702156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:39.758028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:39.757995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:39.841863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:39.841823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:39.940331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:39.940295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:40.040282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:40.040246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:40.387696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:40.387606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:58.328232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:58.328165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:58.565161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:58.565125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:58.996272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:58.996237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:59.271627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.271521 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:48:59.272006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.271991 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1772d450-ce8c-4980-a8c9-fed084bfec94" containerName="revoke-keys" Sep 14 01:48:59.272050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272008 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1772d450-ce8c-4980-a8c9-fed084bfec94" containerName="revoke-keys" Sep 14 01:48:59.272050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272017 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:48:59.272050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272023 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:48:59.272050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272047 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" containerName="payload-processing" Sep 14 01:48:59.272203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272080 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" containerName="payload-processing" Sep 14 01:48:59.272203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272164 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:48:59.272203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272173 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1772d450-ce8c-4980-a8c9-fed084bfec94" containerName="revoke-keys" Sep 14 01:48:59.272203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.272179 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac8f0d6a-3541-4a40-8aa6-c94b4a4b5abb" containerName="payload-processing" Sep 14 01:48:59.276874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.276854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.279380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.279361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7aab97-openshift-default-dockercfg-9bjsq\"" Sep 14 01:48:59.287336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.287314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:48:59.292845 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.292820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:59.329645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7rk\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.330028 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.329962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.430914 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.430879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.430928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.430956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.430994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431122 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7rk\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431392 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431366 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431440 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.431623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.431560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.432041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.432022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.433416 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.433392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.433712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.433692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.439306 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.439285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7rk\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.439385 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.439288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token\") pod \"e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.452813 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.452769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:48:59.589404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.589302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:48:59.868034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.867981 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:48:59.869142 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:48:59.869114 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38254b2a_2424_4212_ac86_53db9e15e8d3.slice/crio-6ca9034a5ee120a593cfec2d134174b3619e7802e48c00c988a00207eebc8d8e WatchSource:0}: Error finding container 6ca9034a5ee120a593cfec2d134174b3619e7802e48c00c988a00207eebc8d8e: Status 404 returned error can't find the container with id 6ca9034a5ee120a593cfec2d134174b3619e7802e48c00c988a00207eebc8d8e Sep 14 01:48:59.871516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.871482 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:48:59.871605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.871561 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:48:59.871605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:48:59.871587 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:49:00.210597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:00.210511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerStarted","Data":"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853"} Sep 14 01:49:00.210597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:00.210573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerStarted","Data":"6ca9034a5ee120a593cfec2d134174b3619e7802e48c00c988a00207eebc8d8e"} Sep 14 01:49:00.230294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:00.230236 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podStartSLOduration=1.230216038 podStartE2EDuration="1.230216038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:49:00.228325798 +0000 UTC m=+1085.461691067" watchObservedRunningTime="2026-09-14 01:49:00.230216038 +0000 UTC m=+1085.463581296" Sep 14 01:49:00.590305 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:00.590268 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:00.595595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:00.595571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:01.215957 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:01.215928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:01.216783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:01.216762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:01.452364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:01.452323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:02.235452 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:02.235414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:02.323416 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:02.323388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:02.997642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:02.997612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:03.860394 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:03.860352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:04.525565 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:04.525525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:05.144682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:05.144653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:05.245915 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:05.245876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:05.345864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:05.345829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:05.672604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:05.672573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:05.769582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:05.769540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:06.301149 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:06.301115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:08.611542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:08.611496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:11.833011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:11.832967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:11.896850 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:11.896813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:11.976192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:11.976154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:12.073256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:12.073215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:25.366646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.366565 2579 generic.go:356] "Generic (PLEG): container finished" podID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerID="1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853" exitCode=0 Sep 14 01:49:25.366646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.366636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerDied","Data":"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853"} Sep 14 01:49:25.367112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.367053 2579 scope.go:117] "RemoveContainer" containerID="1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853" Sep 14 01:49:25.368260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.368234 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:49:25.368316 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.368303 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:49:25.368353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:25.368335 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:49:26.374332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:26.374294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerStarted","Data":"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70"} Sep 14 01:49:26.590520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:26.590485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:26.591676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:26.591652 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:26.591729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:26.591714 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:27.589839 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:27.589808 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:27.590240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:27.589872 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:27.859895 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:27.859805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:27.959941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:27.959887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:28.589716 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:28.589677 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:28.589928 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:28.589735 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:28.646452 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:28.646405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:29.251246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:29.251215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:29.590193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:29.590105 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:29.590193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:29.590162 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:29.865308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:29.865212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:30.502015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:30.501980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:30.590709 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:30.590673 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:30.591100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:30.590730 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:31.157817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:31.157769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:31.216628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:31.216583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:31.590524 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:31.590490 2579 patch_prober.go:28] interesting pod/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 14 01:49:31.590800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:31.590546 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 14 01:49:32.159786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:32.159755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:32.594256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:32.594226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:32.595256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:32.595236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:49:33.596867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:33.596809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:33.703301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:33.703255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:33.797799 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:33.797768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:44.246997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:44.246959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:45.009394 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:45.009354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:50.508493 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:50.508457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:50.567715 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:50.567679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:50.644853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:50.644815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:50.744937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:50.744897 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:50.848401 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:50.848315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:51.395284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:51.395247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:51.495169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:51.495132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:51.595699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:51.595661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:53.059663 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:53.059615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:58.682350 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:58.682312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:58.738050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:58.738015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:49:59.339860 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:49:59.339820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:05.092316 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:05.092283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:05.194488 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:05.194446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:05.320347 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:05.320305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:05.625440 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:05.625401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:05.720584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:05.720547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:10.723891 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:10.723847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:11.015669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:11.015633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:11.107924 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:11.107888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:11.537378 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:11.537344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:12.441131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:12.441098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:13.069983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:13.069945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:13.714240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:13.714201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:14.437192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:14.437155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:14.538295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:14.538261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:14.703006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:14.702908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:14.880729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:14.880695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:14.977896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:14.977788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:15.009139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.009104 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:15.009554 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.009541 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:50:15.009615 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.009556 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:50:15.009653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.009647 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e57041dc-ae03-4a50-a21f-2d47ad840690" containerName="istio-proxy" Sep 14 01:50:15.013107 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.013086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.015439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.015414 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-287e139e-dockercfg-r5dbw\"" Sep 14 01:50:15.015576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.015441 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-287e139e\"" Sep 14 01:50:15.022232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.022151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:15.112497 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.112454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.112699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.112539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcx4n\" (UniqueName: \"kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.213855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.213823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.214079 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.213878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcx4n\" (UniqueName: \"kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.214554 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.214529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.221798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.221774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcx4n\" (UniqueName: \"kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n\") pod \"payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.326108 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.326053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:15.596772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.596742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:15.598788 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:50:15.598752 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cceb676_cc86_4456_a143_7ce158309b18.slice/crio-a20b0f9a6e1a11f05a656638eb71548bf28affa453bae74b9696e76665d11fd3 WatchSource:0}: Error finding container a20b0f9a6e1a11f05a656638eb71548bf28affa453bae74b9696e76665d11fd3: Status 404 returned error can't find the container with id a20b0f9a6e1a11f05a656638eb71548bf28affa453bae74b9696e76665d11fd3 Sep 14 01:50:15.658307 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:15.658275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" event={"ID":"0cceb676-cc86-4456-a143-7ce158309b18","Type":"ContainerStarted","Data":"a20b0f9a6e1a11f05a656638eb71548bf28affa453bae74b9696e76665d11fd3"} Sep 14 01:50:16.666049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:16.666012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" event={"ID":"0cceb676-cc86-4456-a143-7ce158309b18","Type":"ContainerStarted","Data":"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a"} Sep 14 01:50:16.666478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:16.666112 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:16.681858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:16.681806 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" podStartSLOduration=2.681789698 podStartE2EDuration="2.681789698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:50:16.680938569 +0000 UTC m=+1161.914303851" watchObservedRunningTime="2026-09-14 01:50:16.681789698 +0000 UTC m=+1161.915154970" Sep 14 01:50:27.674703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:27.674625 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:29.119088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:29.119034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:33.242035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:33.241975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:37.151020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:37.150980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:38.038535 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:38.038497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:38.654650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:38.654619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:39.436536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:39.436500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:40.152479 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:40.152442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:40.863342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:40.863311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:41.090953 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:41.090919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:41.658121 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:41.658086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:41.758929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:41.758892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:41.860360 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:41.860322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:42.160314 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:42.160278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:47.687384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:47.687339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:47.789101 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:47.789040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:48.023952 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.023911 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q"] Sep 14 01:50:48.027607 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.027589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:48.034746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.034633 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:50:48.036024 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.035996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:50:48.038695 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.038666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q"] Sep 14 01:50:48.151246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.151209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-287e139e-ctf7q\" (UID: \"e36c5f52-0eae-4c63-b375-c65159c11a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:48.252622 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.252577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-287e139e-ctf7q\" (UID: \"e36c5f52-0eae-4c63-b375-c65159c11a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:48.253261 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.253238 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-287e139e-ctf7q\" (UID: \"e36c5f52-0eae-4c63-b375-c65159c11a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:48.347935 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.347850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:48.624726 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.624702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q"] Sep 14 01:50:48.627634 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:50:48.627606 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36c5f52_0eae_4c63_b375_c65159c11a76.slice/crio-caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3 WatchSource:0}: Error finding container caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3: Status 404 returned error can't find the container with id caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3 Sep 14 01:50:48.749360 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.749322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:48.856602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:48.856569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" event={"ID":"e36c5f52-0eae-4c63-b375-c65159c11a76","Type":"ContainerStarted","Data":"caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3"} Sep 14 01:50:49.535438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:49.535358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:49.864332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:49.864299 2579 generic.go:356] "Generic (PLEG): container finished" podID="e36c5f52-0eae-4c63-b375-c65159c11a76" containerID="d194d0421e064ec15c9bec54b37394752fa74f4233d5cc44289902d210a149f1" exitCode=0 Sep 14 01:50:49.864332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:49.864333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" event={"ID":"e36c5f52-0eae-4c63-b375-c65159c11a76","Type":"ContainerDied","Data":"d194d0421e064ec15c9bec54b37394752fa74f4233d5cc44289902d210a149f1"} Sep 14 01:50:50.164423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:50.164329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:50.827278 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:50.827232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:51.025519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.025486 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:51.076618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.076589 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca\") pod \"e36c5f52-0eae-4c63-b375-c65159c11a76\" (UID: \"e36c5f52-0eae-4c63-b375-c65159c11a76\") " Sep 14 01:50:51.076953 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.076930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e36c5f52-0eae-4c63-b375-c65159c11a76" (UID: "e36c5f52-0eae-4c63-b375-c65159c11a76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:50:51.177536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.177453 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36c5f52-0eae-4c63-b375-c65159c11a76-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:50:51.486403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.486367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:51.877410 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.877318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" event={"ID":"e36c5f52-0eae-4c63-b375-c65159c11a76","Type":"ContainerDied","Data":"caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3"} Sep 14 01:50:51.877410 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.877333 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q" Sep 14 01:50:51.877410 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:51.877365 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caeba479f264004a588123b018a03977bbc123b501f674f16e719d8f1901dac3" Sep 14 01:50:52.128971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:52.128876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:52.926586 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:52.926546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:55.235147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.235113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:50:55.236482 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.236464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:50:55.308029 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.307996 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:50:55.308787 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.308759 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36c5f52-0eae-4c63-b375-c65159c11a76" containerName="revoke-keys" Sep 14 01:50:55.308787 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.308787 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36c5f52-0eae-4c63-b375-c65159c11a76" containerName="revoke-keys" Sep 14 01:50:55.308972 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.308939 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36c5f52-0eae-4c63-b375-c65159c11a76" containerName="revoke-keys" Sep 14 01:50:55.312458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.312436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.324479 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.324451 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7aab97\"/\"openshift-service-ca.crt\"" Sep 14 01:50:55.324668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.324647 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7aab97\"/\"default-dockercfg-b2j54\"" Sep 14 01:50:55.324840 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.324548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7aab97\"/\"explicit-model-7aab97-kserve-self-signed-certs\"" Sep 14 01:50:55.325670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.325649 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7aab97\"/\"kube-root-ca.crt\"" Sep 14 01:50:55.327634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.327580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:55.328654 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.328628 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:50:55.424494 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.424666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfmg\" (UniqueName: \"kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.424666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.424666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.424666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.424666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.424619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.525905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.525905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfmg\" (UniqueName: \"kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.525905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526236 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526236 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526236 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.525975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.526283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.526309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.526491 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.526409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.528293 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.528270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.528419 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.528400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.534089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.534068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfmg\" (UniqueName: \"kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg\") pod \"explicit-model-7aab97-kserve-8644647b59-q24js\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.634662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.634626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:50:55.910463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:55.910433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:50:55.911613 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:50:55.911588 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5c39641_dcc2_4af4_874f_ab7f843e1fdd.slice/crio-a652976eb677d17fbfa6164630362b3e18a00a463c22db87cb1d9175e82b107c WatchSource:0}: Error finding container a652976eb677d17fbfa6164630362b3e18a00a463c22db87cb1d9175e82b107c: Status 404 returned error can't find the container with id a652976eb677d17fbfa6164630362b3e18a00a463c22db87cb1d9175e82b107c Sep 14 01:50:56.906896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:56.906856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" event={"ID":"a5c39641-dcc2-4af4-874f-ab7f843e1fdd","Type":"ContainerStarted","Data":"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46"} Sep 14 01:50:56.907365 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:56.906909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" event={"ID":"a5c39641-dcc2-4af4-874f-ab7f843e1fdd","Type":"ContainerStarted","Data":"a652976eb677d17fbfa6164630362b3e18a00a463c22db87cb1d9175e82b107c"} Sep 14 01:50:56.925113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:56.925050 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" podStartSLOduration=1.925035488 podStartE2EDuration="1.925035488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:50:56.922035563 +0000 UTC m=+1202.155400844" watchObservedRunningTime="2026-09-14 01:50:56.925035488 +0000 UTC m=+1202.158400774" Sep 14 01:50:58.440529 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.440490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:58.463043 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.463009 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:58.463298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.463271 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" podUID="0cceb676-cc86-4456-a143-7ce158309b18" containerName="payload-processing" containerID="cri-o://361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a" gracePeriod=30 Sep 14 01:50:58.538526 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.538493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:58.615557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.615496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:58.615880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.615862 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q"] Sep 14 01:50:58.618951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.618925 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-287e139e-ctf7q"] Sep 14 01:50:58.669548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.669504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:58.739133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.739105 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:58.860536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.860502 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume\") pod \"0cceb676-cc86-4456-a143-7ce158309b18\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " Sep 14 01:50:58.860696 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.860559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcx4n\" (UniqueName: \"kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n\") pod \"0cceb676-cc86-4456-a143-7ce158309b18\" (UID: \"0cceb676-cc86-4456-a143-7ce158309b18\") " Sep 14 01:50:58.860843 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.860821 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cceb676-cc86-4456-a143-7ce158309b18" (UID: "0cceb676-cc86-4456-a143-7ce158309b18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:50:58.862525 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.862499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n" (OuterVolumeSpecName: "kube-api-access-zcx4n") pod "0cceb676-cc86-4456-a143-7ce158309b18" (UID: "0cceb676-cc86-4456-a143-7ce158309b18"). InnerVolumeSpecName "kube-api-access-zcx4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:50:58.920904 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.920873 2579 generic.go:356] "Generic (PLEG): container finished" podID="0cceb676-cc86-4456-a143-7ce158309b18" containerID="361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a" exitCode=0 Sep 14 01:50:58.921119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.920922 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" Sep 14 01:50:58.921119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.920956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" event={"ID":"0cceb676-cc86-4456-a143-7ce158309b18","Type":"ContainerDied","Data":"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a"} Sep 14 01:50:58.921119 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.921006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9" event={"ID":"0cceb676-cc86-4456-a143-7ce158309b18","Type":"ContainerDied","Data":"a20b0f9a6e1a11f05a656638eb71548bf28affa453bae74b9696e76665d11fd3"} Sep 14 01:50:58.922274 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.921029 2579 scope.go:117] "RemoveContainer" containerID="361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a" Sep 14 01:50:58.931734 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.931683 2579 scope.go:117] "RemoveContainer" containerID="361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a" Sep 14 01:50:58.932007 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:50:58.931989 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a\": container with ID starting with 361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a not found: ID does not exist" containerID="361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a" Sep 14 01:50:58.932115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.932017 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a"} err="failed to get container status \"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a\": rpc error: code = NotFound desc = could not find container \"361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a\": container with ID starting with 361acf0447eca98971edcc31e95f0e4d22fa6995957862f0cfc2e323924c118a not found: ID does not exist" Sep 14 01:50:58.943694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.943669 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:58.948135 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.948110 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-287e139e-958d5ccbf-d98c9"] Sep 14 01:50:58.962291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.962230 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cceb676-cc86-4456-a143-7ce158309b18-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:50:58.962291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.962253 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcx4n\" (UniqueName: \"kubernetes.io/projected/0cceb676-cc86-4456-a143-7ce158309b18-kube-api-access-zcx4n\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:50:58.969937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:58.969913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:59.125423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:59.125385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:50:59.211329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:59.211298 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cceb676-cc86-4456-a143-7ce158309b18" path="/var/lib/kubelet/pods/0cceb676-cc86-4456-a143-7ce158309b18/volumes" Sep 14 01:50:59.211647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:59.211635 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36c5f52-0eae-4c63-b375-c65159c11a76" path="/var/lib/kubelet/pods/e36c5f52-0eae-4c63-b375-c65159c11a76/volumes" Sep 14 01:50:59.752699 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:50:59.752656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:00.546263 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:00.546229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:01.142687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:01.142655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:01.902635 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:01.902599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:02.530202 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:02.530167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:04.659555 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:04.659518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:04.719961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:04.719927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:04.994576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:04.994543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:05.256346 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.256251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:05.356077 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.356039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:05.635091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.634974 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:51:05.635091 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.635034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:51:05.647683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.647654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:51:05.976647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:05.976561 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:51:06.692304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:06.692270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:11.103987 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:11.103947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:11.160402 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:11.160367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:11.239341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:11.239304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:11.341539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:11.341508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:11.439468 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:11.439373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:15.648320 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:15.648287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:16.983096 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:16.983066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:17.083887 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:17.083852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:17.909506 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:17.909468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:18.222475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:18.222379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:18.550880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:18.550844 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:20.654034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:20.654000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:21.352258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:21.352222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:21.445134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:21.445101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:22.137902 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:22.137859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:22.943495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:22.943461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:23.636260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:23.636229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:24.240698 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:24.240661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:24.938261 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:24.938229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:25.040237 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.040193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:25.230192 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.230147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:25.382985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.382952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:25.470243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.470204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:25.514290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.514208 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:51:25.514740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.514718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cceb676-cc86-4456-a143-7ce158309b18" containerName="payload-processing" Sep 14 01:51:25.514740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.514738 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cceb676-cc86-4456-a143-7ce158309b18" containerName="payload-processing" Sep 14 01:51:25.514943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.514872 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cceb676-cc86-4456-a143-7ce158309b18" containerName="payload-processing" Sep 14 01:51:25.518048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.518027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.520831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.520809 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-38e192\"" Sep 14 01:51:25.520966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.520811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-38e192-dockercfg-rc89l\"" Sep 14 01:51:25.527201 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.527179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:51:25.641598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.641558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c746p\" (UniqueName: \"kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.641765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.641602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.742295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.742257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c746p\" (UniqueName: \"kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.742475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.742312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.743044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.743023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.749370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.749350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c746p\" (UniqueName: \"kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p\") pod \"payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:25.830295 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:25.830203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:26.099421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:26.099397 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:51:26.102366 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:51:26.102334 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9affd72e_8204_4929_954e_5918db2cebe1.slice/crio-d650214bd5f84408716b1c9b5c58aabd1e8191b00bab9cd1658606de1b3368a5 WatchSource:0}: Error finding container d650214bd5f84408716b1c9b5c58aabd1e8191b00bab9cd1658606de1b3368a5: Status 404 returned error can't find the container with id d650214bd5f84408716b1c9b5c58aabd1e8191b00bab9cd1658606de1b3368a5 Sep 14 01:51:27.091143 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:27.091103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" event={"ID":"9affd72e-8204-4929-954e-5918db2cebe1","Type":"ContainerStarted","Data":"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e"} Sep 14 01:51:27.091319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:27.091158 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" event={"ID":"9affd72e-8204-4929-954e-5918db2cebe1","Type":"ContainerStarted","Data":"d650214bd5f84408716b1c9b5c58aabd1e8191b00bab9cd1658606de1b3368a5"} Sep 14 01:51:27.091319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:27.091204 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:27.108210 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:27.108150 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" podStartSLOduration=2.10812854 podStartE2EDuration="2.10812854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:27.106273083 +0000 UTC m=+1232.339638364" watchObservedRunningTime="2026-09-14 01:51:27.10812854 +0000 UTC m=+1232.341493802" Sep 14 01:51:30.080436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.080399 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242"] Sep 14 01:51:30.084291 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.084275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:30.086478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.086456 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:51:30.086614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.086492 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:51:30.090899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.090812 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242"] Sep 14 01:51:30.185915 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.185872 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7aab97-d4242\" (UID: \"043b8247-c834-41b7-b612-f8c08c27774c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:30.287161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.287121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7aab97-d4242\" (UID: \"043b8247-c834-41b7-b612-f8c08c27774c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:30.287739 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.287720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-7aab97-d4242\" (UID: \"043b8247-c834-41b7-b612-f8c08c27774c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:30.357912 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.357822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:30.398278 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.398250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:30.675757 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:30.675726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242"] Sep 14 01:51:31.118377 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:31.118332 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" event={"ID":"043b8247-c834-41b7-b612-f8c08c27774c","Type":"ContainerStarted","Data":"3521cca09df6de25b8d8a89fa18c0eddb34205b0305a77fb4bced8acadb40ed7"} Sep 14 01:51:31.146966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:31.146933 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:31.748227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:31.748191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:32.125709 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:32.125676 2579 generic.go:356] "Generic (PLEG): container finished" podID="043b8247-c834-41b7-b612-f8c08c27774c" containerID="a446650019c559af5ef34a3ef1718debd55b864ed4ccef039fd1d0c23a13bf77" exitCode=0 Sep 14 01:51:32.126140 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:32.125723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" event={"ID":"043b8247-c834-41b7-b612-f8c08c27774c","Type":"ContainerDied","Data":"a446650019c559af5ef34a3ef1718debd55b864ed4ccef039fd1d0c23a13bf77"} Sep 14 01:51:32.446240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:32.446155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:33.128622 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.128579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:33.273628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.273606 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:33.418842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.418743 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca\") pod \"043b8247-c834-41b7-b612-f8c08c27774c\" (UID: \"043b8247-c834-41b7-b612-f8c08c27774c\") " Sep 14 01:51:33.419159 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.419129 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "043b8247-c834-41b7-b612-f8c08c27774c" (UID: "043b8247-c834-41b7-b612-f8c08c27774c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:51:33.520579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.520542 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043b8247-c834-41b7-b612-f8c08c27774c-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:51:33.740510 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:33.740471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:34.139519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:34.139422 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" Sep 14 01:51:34.139519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:34.139460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242" event={"ID":"043b8247-c834-41b7-b612-f8c08c27774c","Type":"ContainerDied","Data":"3521cca09df6de25b8d8a89fa18c0eddb34205b0305a77fb4bced8acadb40ed7"} Sep 14 01:51:34.139954 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:34.139553 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3521cca09df6de25b8d8a89fa18c0eddb34205b0305a77fb4bced8acadb40ed7" Sep 14 01:51:34.536232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:34.536195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:35.142658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:35.142620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:35.753763 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:35.753731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:37.549345 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:37.549310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:38.100035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:38.099999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:51:40.256086 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:40.256027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:40.841231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:40.841197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:40.908951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:40.908913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:40.961900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:40.961855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:41.057443 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.057400 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242"] Sep 14 01:51:41.061042 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.061009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:41.061971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.061949 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-7aab97-d4242"] Sep 14 01:51:41.118801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.118700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:41.215669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.215633 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043b8247-c834-41b7-b612-f8c08c27774c" path="/var/lib/kubelet/pods/043b8247-c834-41b7-b612-f8c08c27774c/volumes" Sep 14 01:51:41.420078 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.419974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:41.558723 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:41.558683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:42.548147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:42.548111 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:43.264308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:43.264273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:44.032629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:44.032588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:44.631296 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:44.631248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:45.443389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:45.443335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:46.064896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.064860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:46.445097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.445004 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:51:46.446664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.446628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:46.518833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.518793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:46.717246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.717162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:46.952897 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:46.952865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:47.032433 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:47.032390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:47.863473 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:47.863443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:50.023355 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:50.023319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:50.561711 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:50.561677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:50.654017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:50.653979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:50.748213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:50.748169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:50.816594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:50.816502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:51.741210 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:51.741162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:52.458254 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:52.458214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:53.233148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:53.233105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:53.910941 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:53.910899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:54.563892 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:54.563853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:54.620039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:54.620003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:54.903607 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:54.903519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:55.105668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.105632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:55.111023 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.110998 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:51:55.111509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.111492 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043b8247-c834-41b7-b612-f8c08c27774c" containerName="revoke-keys" Sep 14 01:51:55.111509 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.111506 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="043b8247-c834-41b7-b612-f8c08c27774c" containerName="revoke-keys" Sep 14 01:51:55.111664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.111576 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="043b8247-c834-41b7-b612-f8c08c27774c" containerName="revoke-keys" Sep 14 01:51:55.114613 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.114594 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.117190 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.117167 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ff93e\"" Sep 14 01:51:55.117294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.117219 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7ff93e-dockercfg-kr2gf\"" Sep 14 01:51:55.125487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.125459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:51:55.144707 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.144672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.144886 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.144765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9jx\" (UniqueName: \"kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.173308 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.173230 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:55.245686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.245653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9jx\" (UniqueName: \"kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.245854 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.245758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.246472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.246452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.253029 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.252998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9jx\" (UniqueName: \"kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx\") pod \"payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.425934 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.425837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:55.698777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.698751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:51:55.700736 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:51:55.700710 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c76e55_4873_4a58_9ba5_c9cffedc1798.slice/crio-ff45309524d3d254278917b5d5cea8cb03c300278eb4f373b659b192fd89400f WatchSource:0}: Error finding container ff45309524d3d254278917b5d5cea8cb03c300278eb4f373b659b192fd89400f: Status 404 returned error can't find the container with id ff45309524d3d254278917b5d5cea8cb03c300278eb4f373b659b192fd89400f Sep 14 01:51:55.835080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:55.835024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:51:56.293764 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:56.293726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" event={"ID":"22c76e55-4873-4a58-9ba5-c9cffedc1798","Type":"ContainerStarted","Data":"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f"} Sep 14 01:51:56.293957 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:56.293785 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" event={"ID":"22c76e55-4873-4a58-9ba5-c9cffedc1798","Type":"ContainerStarted","Data":"ff45309524d3d254278917b5d5cea8cb03c300278eb4f373b659b192fd89400f"} Sep 14 01:51:56.293957 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:56.293816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:51:56.310193 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:51:56.310134 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" podStartSLOduration=1.310111782 podStartE2EDuration="1.310111782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:56.307853349 +0000 UTC m=+1261.541218629" watchObservedRunningTime="2026-09-14 01:51:56.310111782 +0000 UTC m=+1261.543477041" Sep 14 01:52:01.427650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.427615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:01.465002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.464963 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" podUID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" containerName="main" containerID="cri-o://8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46" gracePeriod=45 Sep 14 01:52:01.730797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.730771 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:52:01.810959 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.810927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwfmg\" (UniqueName: \"kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.810959 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.810968 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.811242 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.811242 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811178 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.811242 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811221 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.811422 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache\") pod \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\" (UID: \"a5c39641-dcc2-4af4-874f-ab7f843e1fdd\") " Sep 14 01:52:01.811500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811468 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:01.811687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811640 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:01.811758 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811728 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tmp-dir\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:01.811820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.811787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home" (OuterVolumeSpecName: "home") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:01.813450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.813421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:52:01.813558 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.813429 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm" (OuterVolumeSpecName: "dshm") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:01.813558 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.813519 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg" (OuterVolumeSpecName: "kube-api-access-wwfmg") pod "a5c39641-dcc2-4af4-874f-ab7f843e1fdd" (UID: "a5c39641-dcc2-4af4-874f-ab7f843e1fdd"). InnerVolumeSpecName "kube-api-access-wwfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:52:01.913281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.913244 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-home\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:01.913281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.913275 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-model-cache\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:01.913495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.913300 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfmg\" (UniqueName: \"kubernetes.io/projected/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-kube-api-access-wwfmg\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:01.913495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.913309 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-tls-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:01.913495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:01.913318 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c39641-dcc2-4af4-874f-ab7f843e1fdd-dshm\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:02.329328 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.329297 2579 generic.go:356] "Generic (PLEG): container finished" podID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" containerID="8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46" exitCode=0 Sep 14 01:52:02.329546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.329341 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" Sep 14 01:52:02.329546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.329385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" event={"ID":"a5c39641-dcc2-4af4-874f-ab7f843e1fdd","Type":"ContainerDied","Data":"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46"} Sep 14 01:52:02.329546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.329434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js" event={"ID":"a5c39641-dcc2-4af4-874f-ab7f843e1fdd","Type":"ContainerDied","Data":"a652976eb677d17fbfa6164630362b3e18a00a463c22db87cb1d9175e82b107c"} Sep 14 01:52:02.329546 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.329456 2579 scope.go:117] "RemoveContainer" containerID="8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46" Sep 14 01:52:02.339940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.339916 2579 scope.go:117] "RemoveContainer" containerID="8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46" Sep 14 01:52:02.340270 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:52:02.340250 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46\": container with ID starting with 8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46 not found: ID does not exist" containerID="8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46" Sep 14 01:52:02.340332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.340281 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46"} err="failed to get container status \"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46\": rpc error: code = NotFound desc = could not find container \"8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46\": container with ID starting with 8b65f01db1af9fd2542d44fde7dd98bb10276e244f715f937434e2b972a8bc46 not found: ID does not exist" Sep 14 01:52:02.350764 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.350736 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:52:02.353076 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:02.353045 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7aab97/explicit-model-7aab97-kserve-8644647b59-q24js"] Sep 14 01:52:03.211235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:03.211194 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" path="/var/lib/kubelet/pods/a5c39641-dcc2-4af4-874f-ab7f843e1fdd/volumes" Sep 14 01:52:07.301451 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:07.301412 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:52:13.160212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:13.160171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:15.367313 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:15.367275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:16.851668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:16.851628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:17.730559 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:17.730523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:18.551753 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:18.551712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:19.243858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:19.243828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:20.069618 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:20.069580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:20.613330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:20.613289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:20.788324 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:20.788293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:21.643825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:21.643784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:23.388835 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:23.388793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:23.612952 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:23.612915 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:52:23.613206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:23.613178 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" containerID="cri-o://3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70" gracePeriod=30 Sep 14 01:52:23.627876 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:23.627845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:24.051399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.051363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:24.310147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.310069 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:52:24.310623 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.310605 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" containerName="main" Sep 14 01:52:24.310732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.310626 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" containerName="main" Sep 14 01:52:24.310785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.310769 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c39641-dcc2-4af4-874f-ab7f843e1fdd" containerName="main" Sep 14 01:52:24.315458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.315434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.317667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.317633 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-06e7b2-openshift-default-dockercfg-lnpbh\"" Sep 14 01:52:24.321902 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.321870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:52:24.336579 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.336538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j54d\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.439997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.440139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.440309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.440170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541540 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541540 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j54d\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541870 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541849 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.541975 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.541947 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.542101 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.542053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.542260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.542148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.542352 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.542203 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.543814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.543787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.543919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.543902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.548825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.548801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j54d\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.549047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.549030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token\") pod \"e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.564752 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.564684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:24.628216 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.628182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:24.903213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.903187 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:52:24.904273 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:52:24.904239 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67561d02_e1e9_44f6_9c0a_7c33e8f71839.slice/crio-1f2e198f32fbd49934fec42dfbcb1dbc0be40792ada904528bf92f6bcaa5c0e4 WatchSource:0}: Error finding container 1f2e198f32fbd49934fec42dfbcb1dbc0be40792ada904528bf92f6bcaa5c0e4: Status 404 returned error can't find the container with id 1f2e198f32fbd49934fec42dfbcb1dbc0be40792ada904528bf92f6bcaa5c0e4 Sep 14 01:52:24.906649 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.906616 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:52:24.906742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.906683 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:52:24.906742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:24.906710 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:52:25.480222 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.480184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" event={"ID":"67561d02-e1e9-44f6-9c0a-7c33e8f71839","Type":"ContainerStarted","Data":"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d"} Sep 14 01:52:25.480604 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.480245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" event={"ID":"67561d02-e1e9-44f6-9c0a-7c33e8f71839","Type":"ContainerStarted","Data":"1f2e198f32fbd49934fec42dfbcb1dbc0be40792ada904528bf92f6bcaa5c0e4"} Sep 14 01:52:25.498218 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.498170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" podStartSLOduration=1.498133167 podStartE2EDuration="1.498133167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:52:25.496485604 +0000 UTC m=+1290.729850882" watchObservedRunningTime="2026-09-14 01:52:25.498133167 +0000 UTC m=+1290.731498424" Sep 14 01:52:25.629760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.629722 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:25.634894 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.634868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:25.771341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.771251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:25.851697 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.851655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:25.966405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:25.966366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:26.486754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:26.486717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:26.487825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:26.487804 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:52:28.873472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.873443 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:52:28.884991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.884928 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.884991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.884962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.884991 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.884981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885090 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885117 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885162 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885210 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm7rk\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk\") pod \"38254b2a-2424-4212-ac86-53db9e15e8d3\" (UID: \"38254b2a-2424-4212-ac86-53db9e15e8d3\") " Sep 14 01:52:28.885279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:28.885603 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885576 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:28.885682 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885658 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:28.885864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885833 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.885864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885858 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.886044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885873 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.886044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885925 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:28.886044 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.885953 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:52:28.887337 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.887313 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:52:28.887637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.887615 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:52:28.887715 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.887645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk" (OuterVolumeSpecName: "kube-api-access-nm7rk") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "kube-api-access-nm7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:52:28.887715 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.887689 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "38254b2a-2424-4212-ac86-53db9e15e8d3" (UID: "38254b2a-2424-4212-ac86-53db9e15e8d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:52:28.986600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986567 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.986600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986595 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.986600 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986605 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38254b2a-2424-4212-ac86-53db9e15e8d3-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.986821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986613 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7rk\" (UniqueName: \"kubernetes.io/projected/38254b2a-2424-4212-ac86-53db9e15e8d3-kube-api-access-nm7rk\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.986821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986623 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:28.986821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:28.986632 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38254b2a-2424-4212-ac86-53db9e15e8d3-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:52:29.513569 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.513538 2579 generic.go:356] "Generic (PLEG): container finished" podID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerID="3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70" exitCode=0 Sep 14 01:52:29.513755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.513579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerDied","Data":"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70"} Sep 14 01:52:29.513755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.513589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" Sep 14 01:52:29.513755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.513619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn" event={"ID":"38254b2a-2424-4212-ac86-53db9e15e8d3","Type":"ContainerDied","Data":"6ca9034a5ee120a593cfec2d134174b3619e7802e48c00c988a00207eebc8d8e"} Sep 14 01:52:29.513755 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.513641 2579 scope.go:117] "RemoveContainer" containerID="3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70" Sep 14 01:52:29.523691 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.523659 2579 scope.go:117] "RemoveContainer" containerID="1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853" Sep 14 01:52:29.530209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.530180 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:52:29.532979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.532952 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7aab97-openshift-default-7f96b887cb-j7zxn"] Sep 14 01:52:29.533432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.533413 2579 scope.go:117] "RemoveContainer" containerID="3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70" Sep 14 01:52:29.533729 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:52:29.533709 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70\": container with ID starting with 3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70 not found: ID does not exist" containerID="3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70" Sep 14 01:52:29.533804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.533741 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70"} err="failed to get container status \"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70\": rpc error: code = NotFound desc = could not find container \"3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70\": container with ID starting with 3e6c89b82feb56d5b17ff4226c53a2d79ac3552aed0810b8c379412b72f1bd70 not found: ID does not exist" Sep 14 01:52:29.533804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.533764 2579 scope.go:117] "RemoveContainer" containerID="1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853" Sep 14 01:52:29.534031 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:52:29.534016 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853\": container with ID starting with 1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853 not found: ID does not exist" containerID="1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853" Sep 14 01:52:29.534142 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.534038 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853"} err="failed to get container status \"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853\": rpc error: code = NotFound desc = could not find container \"1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853\": container with ID starting with 1b4624ef37861b2869b02563fa93e01fe075f494b54bc851e95959d8caf14853 not found: ID does not exist" Sep 14 01:52:29.595982 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.595949 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:52:29.596436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596422 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.596481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596438 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.596481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596448 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.596481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596455 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.596571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596537 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.596571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.596548 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" containerName="istio-proxy" Sep 14 01:52:29.601154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.601139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.603587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.603523 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"kube-root-ca.crt\"" Sep 14 01:52:29.603703 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.603616 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"openshift-service-ca.crt\"" Sep 14 01:52:29.603863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.603838 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"default-dockercfg-j965m\"" Sep 14 01:52:29.604000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.603855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"e2e-conflict-a-9e04456b-kserve-self-signed-certs\"" Sep 14 01:52:29.606644 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.606620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:52:29.692364 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.692492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.692492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.692492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.692492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txrv\" (UniqueName: \"kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.692678 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.692554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.746324 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.746286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:29.794000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.793931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.793971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txrv\" (UniqueName: \"kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.794627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.794514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.796331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.796311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.796463 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.796446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.801288 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.801267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txrv\" (UniqueName: \"kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv\") pod \"e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:29.898881 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.898849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:29.912381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:29.912358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:30.028530 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.028498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:30.193945 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.193913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:52:30.196130 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:52:30.196104 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41601062_844f_4398_9e74_ec159bb6523e.slice/crio-2c31fa4c5c0bb0c88b1469321a0ee11bf7074b0286feeec94216b6001ba2b2be WatchSource:0}: Error finding container 2c31fa4c5c0bb0c88b1469321a0ee11bf7074b0286feeec94216b6001ba2b2be: Status 404 returned error can't find the container with id 2c31fa4c5c0bb0c88b1469321a0ee11bf7074b0286feeec94216b6001ba2b2be Sep 14 01:52:30.430713 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.430670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:30.523602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.523566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" event={"ID":"41601062-844f-4398-9e74-ec159bb6523e","Type":"ContainerStarted","Data":"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd"} Sep 14 01:52:30.523783 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.523611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" event={"ID":"41601062-844f-4398-9e74-ec159bb6523e","Type":"ContainerStarted","Data":"2c31fa4c5c0bb0c88b1469321a0ee11bf7074b0286feeec94216b6001ba2b2be"} Sep 14 01:52:30.539661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:30.539608 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" podStartSLOduration=1.539592571 podStartE2EDuration="1.539592571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:52:30.537756554 +0000 UTC m=+1295.771121836" watchObservedRunningTime="2026-09-14 01:52:30.539592571 +0000 UTC m=+1295.772957828" Sep 14 01:52:31.211350 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:31.211307 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38254b2a-2424-4212-ac86-53db9e15e8d3" path="/var/lib/kubelet/pods/38254b2a-2424-4212-ac86-53db9e15e8d3/volumes" Sep 14 01:52:32.033147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:32.033105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:32.635033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:32.634991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:32.740330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:32.740300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:32.842097 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:32.842042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:33.547535 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:33.547500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:34.299040 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:34.299000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:35.017969 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:35.017932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:35.644695 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:35.644656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:36.330806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:36.330767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:37.142417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.142384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:37.243478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.243438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:37.576070 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.576019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:37.675363 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.675330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:37.683384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.683354 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:52:37.687414 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.687397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.690135 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.690119 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-000f34a9\"" Sep 14 01:52:37.690231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.690163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-000f34a9-dockercfg-gd9n8\"" Sep 14 01:52:37.694855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.694830 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:52:37.718779 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.718734 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:52:37.722363 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.722341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.729722 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.729698 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:52:37.771844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.771559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlhw\" (UniqueName: \"kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.771844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.771740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.771844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.771800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.771844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.771840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bs4\" (UniqueName: \"kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.873367 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.873286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlhw\" (UniqueName: \"kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.873367 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.873351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.873612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.873381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.873612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.873404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bs4\" (UniqueName: \"kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.874180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.874156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.874283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.874156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:37.880916 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.880892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bs4\" (UniqueName: \"kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4\") pod \"payload-processing-e2e-mt-000f34a9-686c44d485-s2svn\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:37.881006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:37.880919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlhw\" (UniqueName: \"kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw\") pod \"payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:38.000404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.000372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:38.035368 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.035333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:38.272965 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.272938 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:52:38.275083 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:52:38.275031 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d8d736_8d1c_4f81_9d90_7591cc8263a2.slice/crio-374ca5dfc20d19321b42c8d04f9fbca645aca115ad9700ec110a3eeebed2f09a WatchSource:0}: Error finding container 374ca5dfc20d19321b42c8d04f9fbca645aca115ad9700ec110a3eeebed2f09a: Status 404 returned error can't find the container with id 374ca5dfc20d19321b42c8d04f9fbca645aca115ad9700ec110a3eeebed2f09a Sep 14 01:52:38.324336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.324312 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:52:38.325883 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:52:38.325860 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f32d5a1_05e7_45e1_9c96_55cc8fd2f9ac.slice/crio-bf24cc81348b6b999a68a572eb5b41cb8159a8b0f414c1bcc539e918c25e6e11 WatchSource:0}: Error finding container bf24cc81348b6b999a68a572eb5b41cb8159a8b0f414c1bcc539e918c25e6e11: Status 404 returned error can't find the container with id bf24cc81348b6b999a68a572eb5b41cb8159a8b0f414c1bcc539e918c25e6e11 Sep 14 01:52:38.577131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.577029 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" event={"ID":"17d8d736-8d1c-4f81-9d90-7591cc8263a2","Type":"ContainerStarted","Data":"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca"} Sep 14 01:52:38.577131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.577106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" event={"ID":"17d8d736-8d1c-4f81-9d90-7591cc8263a2","Type":"ContainerStarted","Data":"374ca5dfc20d19321b42c8d04f9fbca645aca115ad9700ec110a3eeebed2f09a"} Sep 14 01:52:38.577331 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.577242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:38.578778 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.578751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" event={"ID":"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac","Type":"ContainerStarted","Data":"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336"} Sep 14 01:52:38.578896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.578793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" event={"ID":"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac","Type":"ContainerStarted","Data":"bf24cc81348b6b999a68a572eb5b41cb8159a8b0f414c1bcc539e918c25e6e11"} Sep 14 01:52:38.578896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.578824 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:38.593424 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.593382 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" podStartSLOduration=1.5933669419999998 podStartE2EDuration="1.593366942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:52:38.591528987 +0000 UTC m=+1303.824894266" watchObservedRunningTime="2026-09-14 01:52:38.593366942 +0000 UTC m=+1303.826732198" Sep 14 01:52:38.611564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:38.611515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" podStartSLOduration=1.6115017040000001 podStartE2EDuration="1.611501704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:52:38.609874891 +0000 UTC m=+1303.843240174" watchObservedRunningTime="2026-09-14 01:52:38.611501704 +0000 UTC m=+1303.844866961" Sep 14 01:52:39.913237 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:39.913198 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:39.913625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:39.913252 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:39.931774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:39.931739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:40.602570 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:40.602541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:52:49.587815 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:49.587785 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:52:49.588272 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:49.587870 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:52:55.340106 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:55.340076 2579 scope.go:117] "RemoveContainer" containerID="4fda18b9c7d4154ae8b0a1e59e094a6ac3fd8dd75878345fbeb7d156cedd7e8c" Sep 14 01:52:59.252490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:59.252440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:52:59.949646 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:52:59.949612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:00.625080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:00.625021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:01.570184 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:01.570143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:02.255716 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:02.255678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:02.752341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:02.752306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:03.039981 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:03.039898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:03.772807 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:03.772768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:04.408222 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:04.408164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:05.480015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:05.479972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:08.107980 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:08.107934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:17.514515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.514480 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq"] Sep 14 01:53:17.519668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.519648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:17.522574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.522551 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:53:17.522770 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.522572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:53:17.525163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.525138 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq"] Sep 14 01:53:17.579224 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.579200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq\" (UID: \"63ef15f7-dfa8-43f0-a955-746e70fdd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:17.650412 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.650377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:17.680622 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.680592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq\" (UID: \"63ef15f7-dfa8-43f0-a955-746e70fdd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:17.681258 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.681237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq\" (UID: \"63ef15f7-dfa8-43f0-a955-746e70fdd66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:17.832951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:17.832873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:18.101493 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:18.101462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq"] Sep 14 01:53:18.102848 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:53:18.102822 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ef15f7_dfa8_43f0_a955_746e70fdd66e.slice/crio-71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8 WatchSource:0}: Error finding container 71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8: Status 404 returned error can't find the container with id 71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8 Sep 14 01:53:18.321126 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:18.321096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:18.825138 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:18.825099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" event={"ID":"63ef15f7-dfa8-43f0-a955-746e70fdd66e","Type":"ContainerStarted","Data":"96d3a84cc02754fb4570346a9d1ee937779ce4d78f0fdfd85ceeacbeaf7f9fb3"} Sep 14 01:53:18.825548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:18.825168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" event={"ID":"63ef15f7-dfa8-43f0-a955-746e70fdd66e","Type":"ContainerStarted","Data":"71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8"} Sep 14 01:53:18.840922 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:18.840879 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" podStartSLOduration=1.840864735 podStartE2EDuration="1.840864735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:53:18.839193239 +0000 UTC m=+1344.072558522" watchObservedRunningTime="2026-09-14 01:53:18.840864735 +0000 UTC m=+1344.074230042" Sep 14 01:53:19.020686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:19.020640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:19.649269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:19.649230 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:19.832948 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:19.832913 2579 generic.go:356] "Generic (PLEG): container finished" podID="63ef15f7-dfa8-43f0-a955-746e70fdd66e" containerID="96d3a84cc02754fb4570346a9d1ee937779ce4d78f0fdfd85ceeacbeaf7f9fb3" exitCode=0 Sep 14 01:53:19.833330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:19.833005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" event={"ID":"63ef15f7-dfa8-43f0-a955-746e70fdd66e","Type":"ContainerDied","Data":"96d3a84cc02754fb4570346a9d1ee937779ce4d78f0fdfd85ceeacbeaf7f9fb3"} Sep 14 01:53:20.271434 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:20.271403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:20.977409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:20.977386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:21.042616 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.042573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:21.119421 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.118950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca\") pod \"63ef15f7-dfa8-43f0-a955-746e70fdd66e\" (UID: \"63ef15f7-dfa8-43f0-a955-746e70fdd66e\") " Sep 14 01:53:21.120409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.119768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63ef15f7-dfa8-43f0-a955-746e70fdd66e" (UID: "63ef15f7-dfa8-43f0-a955-746e70fdd66e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:21.120409 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.120375 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ef15f7-dfa8-43f0-a955-746e70fdd66e-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:21.646869 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.646834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:21.846322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.846294 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" Sep 14 01:53:21.846488 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.846334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq" event={"ID":"63ef15f7-dfa8-43f0-a955-746e70fdd66e","Type":"ContainerDied","Data":"71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8"} Sep 14 01:53:21.846488 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:21.846378 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a2bdc58cbc0b7cbd4daf50f3cf43ffbe040f27efe3e80b163461ed85b518b8" Sep 14 01:53:22.263260 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:22.263227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:23.038938 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:23.038901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:23.740761 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:23.740730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:24.707937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:24.707899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:27.636989 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:27.636956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.270336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.270295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.303196 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.303158 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:53:28.303460 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.303426 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" podUID="fba63483-50fb-446e-bbdf-fc621b0f044b" containerName="payload-processing" containerID="cri-o://2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42" gracePeriod=30 Sep 14 01:53:28.368243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.368209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.458251 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.458221 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq"] Sep 14 01:53:28.458923 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.458885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.461749 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.461727 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a36bf3-z2zqq"] Sep 14 01:53:28.510338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.510249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.586284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.586260 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:53:28.696580 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.696545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcl86\" (UniqueName: \"kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86\") pod \"fba63483-50fb-446e-bbdf-fc621b0f044b\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " Sep 14 01:53:28.696951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.696631 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume\") pod \"fba63483-50fb-446e-bbdf-fc621b0f044b\" (UID: \"fba63483-50fb-446e-bbdf-fc621b0f044b\") " Sep 14 01:53:28.696997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.696970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fba63483-50fb-446e-bbdf-fc621b0f044b" (UID: "fba63483-50fb-446e-bbdf-fc621b0f044b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:28.697099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.697079 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba63483-50fb-446e-bbdf-fc621b0f044b-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.698694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.698676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86" (OuterVolumeSpecName: "kube-api-access-rcl86") pod "fba63483-50fb-446e-bbdf-fc621b0f044b" (UID: "fba63483-50fb-446e-bbdf-fc621b0f044b"). InnerVolumeSpecName "kube-api-access-rcl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:28.798299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.798254 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcl86\" (UniqueName: \"kubernetes.io/projected/fba63483-50fb-446e-bbdf-fc621b0f044b-kube-api-access-rcl86\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.869069 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.868969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:28.888208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.888178 2579 generic.go:356] "Generic (PLEG): container finished" podID="fba63483-50fb-446e-bbdf-fc621b0f044b" containerID="2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42" exitCode=0 Sep 14 01:53:28.888353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.888221 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" Sep 14 01:53:28.888353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.888261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" event={"ID":"fba63483-50fb-446e-bbdf-fc621b0f044b","Type":"ContainerDied","Data":"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42"} Sep 14 01:53:28.888353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.888322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg" event={"ID":"fba63483-50fb-446e-bbdf-fc621b0f044b","Type":"ContainerDied","Data":"f4e2f70571ad80f2f0929f73fd20b198d03f9f22c160abafa7d09f150b224f2e"} Sep 14 01:53:28.888353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.888350 2579 scope.go:117] "RemoveContainer" containerID="2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42" Sep 14 01:53:28.898391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.898375 2579 scope.go:117] "RemoveContainer" containerID="2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42" Sep 14 01:53:28.898671 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:53:28.898651 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42\": container with ID starting with 2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42 not found: ID does not exist" containerID="2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42" Sep 14 01:53:28.898742 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.898684 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42"} err="failed to get container status \"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42\": rpc error: code = NotFound desc = could not find container \"2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42\": container with ID starting with 2bc905ae93aa661b6ca1e511bd56f90d689400b78124ee6664f5b17c57436d42 not found: ID does not exist" Sep 14 01:53:28.911309 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.911276 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:53:28.913514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.913494 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a36bf3-86d9747967-98rkg"] Sep 14 01:53:28.928169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:28.928140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:29.210671 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:29.210585 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ef15f7-dfa8-43f0-a955-746e70fdd66e" path="/var/lib/kubelet/pods/63ef15f7-dfa8-43f0-a955-746e70fdd66e/volumes" Sep 14 01:53:29.210973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:29.210958 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba63483-50fb-446e-bbdf-fc621b0f044b" path="/var/lib/kubelet/pods/fba63483-50fb-446e-bbdf-fc621b0f044b/volumes" Sep 14 01:53:29.739866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:29.739835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:30.508164 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:30.508132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:31.072335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:31.072296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:31.845879 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:31.845835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:32.448681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:32.448639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:33.049769 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:33.049718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:34.727744 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:34.727702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:34.778335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:34.778300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:34.990526 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:34.990491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:35.345190 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345113 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz"] Sep 14 01:53:35.345584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345572 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ef15f7-dfa8-43f0-a955-746e70fdd66e" containerName="revoke-keys" Sep 14 01:53:35.345639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345585 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ef15f7-dfa8-43f0-a955-746e70fdd66e" containerName="revoke-keys" Sep 14 01:53:35.345639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345596 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba63483-50fb-446e-bbdf-fc621b0f044b" containerName="payload-processing" Sep 14 01:53:35.345639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345602 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba63483-50fb-446e-bbdf-fc621b0f044b" containerName="payload-processing" Sep 14 01:53:35.345753 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345667 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba63483-50fb-446e-bbdf-fc621b0f044b" containerName="payload-processing" Sep 14 01:53:35.345753 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.345675 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ef15f7-dfa8-43f0-a955-746e70fdd66e" containerName="revoke-keys" Sep 14 01:53:35.350303 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.350286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:35.353780 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.353754 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:53:35.353973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.353845 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:53:35.354584 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.354560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz"] Sep 14 01:53:35.464038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.463996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-321811-qlngz\" (UID: \"4c5569b2-70f3-4958-80f4-a648b8640d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:35.565540 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.565507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-321811-qlngz\" (UID: \"4c5569b2-70f3-4958-80f4-a648b8640d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:35.566102 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.566085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-321811-qlngz\" (UID: \"4c5569b2-70f3-4958-80f4-a648b8640d8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:35.664588 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.664504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:35.939194 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.939170 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz"] Sep 14 01:53:35.941072 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:53:35.941028 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5569b2_70f3_4958_80f4_a648b8640d8b.slice/crio-42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e WatchSource:0}: Error finding container 42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e: Status 404 returned error can't find the container with id 42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e Sep 14 01:53:35.994827 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:35.994799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:36.721123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:36.721097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:36.939733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:36.939698 2579 generic.go:356] "Generic (PLEG): container finished" podID="4c5569b2-70f3-4958-80f4-a648b8640d8b" containerID="281218cd9431aec4a4866ada2f92a20de51694490c35e55b879a4c7d8035f489" exitCode=0 Sep 14 01:53:36.940132 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:36.939765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" event={"ID":"4c5569b2-70f3-4958-80f4-a648b8640d8b","Type":"ContainerDied","Data":"281218cd9431aec4a4866ada2f92a20de51694490c35e55b879a4c7d8035f489"} Sep 14 01:53:36.940132 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:36.939802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" event={"ID":"4c5569b2-70f3-4958-80f4-a648b8640d8b","Type":"ContainerStarted","Data":"42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e"} Sep 14 01:53:37.333658 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:37.333623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:37.437983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:37.437943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:38.033776 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.033738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:38.096040 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.096009 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:38.190410 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.190377 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca\") pod \"4c5569b2-70f3-4958-80f4-a648b8640d8b\" (UID: \"4c5569b2-70f3-4958-80f4-a648b8640d8b\") " Sep 14 01:53:38.190720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.190698 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c5569b2-70f3-4958-80f4-a648b8640d8b" (UID: "4c5569b2-70f3-4958-80f4-a648b8640d8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:38.291154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.291122 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5569b2-70f3-4958-80f4-a648b8640d8b-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:38.743305 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.743265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:38.953265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.953228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" event={"ID":"4c5569b2-70f3-4958-80f4-a648b8640d8b","Type":"ContainerDied","Data":"42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e"} Sep 14 01:53:38.953265 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.953250 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz" Sep 14 01:53:38.953517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:38.953276 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f0ad0492e5011e2834afa5856b5d2d6eace7b8f176d6bce5d500682b91d02e" Sep 14 01:53:39.546909 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:39.546873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:40.175796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:40.175763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:40.313564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:40.313515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:45.941231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:45.941191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:45.950201 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:45.950157 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:53:45.950377 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:45.950353 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" podUID="b228d39c-ba77-4a57-9da4-39070f186b6f" containerName="payload-processing" containerID="cri-o://e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906" gracePeriod=30 Sep 14 01:53:46.040576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.040540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:46.102465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.102415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz"] Sep 14 01:53:46.102827 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.102793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:46.105506 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.105463 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-321811-qlngz"] Sep 14 01:53:46.153962 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.153931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:46.258458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.258418 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:53:46.365587 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.365557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume\") pod \"b228d39c-ba77-4a57-9da4-39070f186b6f\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " Sep 14 01:53:46.365785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.365665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq6qn\" (UniqueName: \"kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn\") pod \"b228d39c-ba77-4a57-9da4-39070f186b6f\" (UID: \"b228d39c-ba77-4a57-9da4-39070f186b6f\") " Sep 14 01:53:46.365983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.365955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b228d39c-ba77-4a57-9da4-39070f186b6f" (UID: "b228d39c-ba77-4a57-9da4-39070f186b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:46.366088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.366072 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b228d39c-ba77-4a57-9da4-39070f186b6f-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:46.367648 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.367630 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn" (OuterVolumeSpecName: "kube-api-access-nq6qn") pod "b228d39c-ba77-4a57-9da4-39070f186b6f" (UID: "b228d39c-ba77-4a57-9da4-39070f186b6f"). InnerVolumeSpecName "kube-api-access-nq6qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:46.463464 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.463423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:46.467607 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.467578 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq6qn\" (UniqueName: \"kubernetes.io/projected/b228d39c-ba77-4a57-9da4-39070f186b6f-kube-api-access-nq6qn\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:53:46.564090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:46.564044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:47.005873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.005839 2579 generic.go:356] "Generic (PLEG): container finished" podID="b228d39c-ba77-4a57-9da4-39070f186b6f" containerID="e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906" exitCode=0 Sep 14 01:53:47.006359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.005877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" event={"ID":"b228d39c-ba77-4a57-9da4-39070f186b6f","Type":"ContainerDied","Data":"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906"} Sep 14 01:53:47.006359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.005885 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" Sep 14 01:53:47.006359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.005926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b" event={"ID":"b228d39c-ba77-4a57-9da4-39070f186b6f","Type":"ContainerDied","Data":"d96784d038d7f71c1b3667e934471f24ad008717e4e403b8849fc552e80ed859"} Sep 14 01:53:47.006359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.005948 2579 scope.go:117] "RemoveContainer" containerID="e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906" Sep 14 01:53:47.017116 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.017092 2579 scope.go:117] "RemoveContainer" containerID="e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906" Sep 14 01:53:47.017393 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:53:47.017374 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906\": container with ID starting with e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906 not found: ID does not exist" containerID="e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906" Sep 14 01:53:47.017468 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.017399 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906"} err="failed to get container status \"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906\": rpc error: code = NotFound desc = could not find container \"e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906\": container with ID starting with e6818412da00d8cb3301537adec5a09916236a014fda8388c2120745329fa906 not found: ID does not exist" Sep 14 01:53:47.030016 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.029992 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:53:47.033311 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.033288 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-321811-c87f9db8c-8hx2b"] Sep 14 01:53:47.183329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.183293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:47.211253 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.211219 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c5569b2-70f3-4958-80f4-a648b8640d8b" path="/var/lib/kubelet/pods/4c5569b2-70f3-4958-80f4-a648b8640d8b/volumes" Sep 14 01:53:47.211597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.211582 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b228d39c-ba77-4a57-9da4-39070f186b6f" path="/var/lib/kubelet/pods/b228d39c-ba77-4a57-9da4-39070f186b6f/volumes" Sep 14 01:53:47.933347 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:47.933315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:48.659666 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:48.659629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:49.201610 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:49.201573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:49.366814 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:49.366778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:50.141457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:50.141417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:51.533990 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:51.533952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:55.371161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:55.371127 2579 scope.go:117] "RemoveContainer" containerID="9b7ec24f4a0de71419df6c3928ca2e9280da9f86622855753231754435856bbd" Sep 14 01:53:55.380116 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:55.380096 2579 scope.go:117] "RemoveContainer" containerID="9827016ce2f98a9625ddfec0cf0674d21b6952c3ce12fe493585ec2833942617" Sep 14 01:53:56.822892 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:56.822849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:57.206541 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:57.206448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:57.430299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:57.430264 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:53:57.430571 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:57.430531 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" containerID="cri-o://ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8" gracePeriod=30 Sep 14 01:53:57.513539 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:57.513499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:58.069868 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:58.069838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:58.385773 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:58.385685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:58.677240 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:58.677147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:59.018224 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:59.018191 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:53:59.018472 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:59.018440 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" podUID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" containerName="istio-proxy" containerID="cri-o://f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d" gracePeriod=30 Sep 14 01:53:59.035610 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:59.035581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:59.235778 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:59.235730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:53:59.346420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:53:59.346342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:00.638297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:00.638262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:01.112551 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:01.112518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:01.254259 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:01.254221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:01.352821 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:01.352782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:02.055403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.055370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:02.692888 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.692819 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:54:02.832400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.832371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:02.841979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.841947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.841983 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842030 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842101 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842230 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:02.842312 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842235 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842316 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:02.842432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842553 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9f5j\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j\") pod \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\" (UID: \"0cfba7ba-4d93-4b93-a7e0-c092f576ed80\") " Sep 14 01:54:02.842690 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:02.842842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842816 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:54:02.842940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842924 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.843000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842946 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.843000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842960 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.843000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.842973 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.843190 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.843025 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:02.844545 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.844512 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:54:02.844545 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.844518 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:02.844671 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.844551 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:54:02.844671 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.844643 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j" (OuterVolumeSpecName: "kube-api-access-z9f5j") pod "0cfba7ba-4d93-4b93-a7e0-c092f576ed80" (UID: "0cfba7ba-4d93-4b93-a7e0-c092f576ed80"). InnerVolumeSpecName "kube-api-access-z9f5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:54:02.944465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.944377 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9f5j\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-kube-api-access-z9f5j\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.944465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.944413 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.944465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.944422 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.944465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.944430 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:02.944465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:02.944439 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfba7ba-4d93-4b93-a7e0-c092f576ed80-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:03.107281 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.107242 2579 generic.go:356] "Generic (PLEG): container finished" podID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerID="ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8" exitCode=0 Sep 14 01:54:03.107676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.107287 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" Sep 14 01:54:03.107676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.107330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerDied","Data":"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8"} Sep 14 01:54:03.107676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.107396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m" event={"ID":"0cfba7ba-4d93-4b93-a7e0-c092f576ed80","Type":"ContainerDied","Data":"bc5afadd49dcee5d2290ced684e42eafaed0a9197c0c03ebed234d03dfcb70d1"} Sep 14 01:54:03.107676 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.107425 2579 scope.go:117] "RemoveContainer" containerID="ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8" Sep 14 01:54:03.118006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.117985 2579 scope.go:117] "RemoveContainer" containerID="8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a" Sep 14 01:54:03.126689 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.126670 2579 scope.go:117] "RemoveContainer" containerID="ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8" Sep 14 01:54:03.126961 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:54:03.126940 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8\": container with ID starting with ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8 not found: ID does not exist" containerID="ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8" Sep 14 01:54:03.127009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.126971 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8"} err="failed to get container status \"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8\": rpc error: code = NotFound desc = could not find container \"ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8\": container with ID starting with ffc287030cdbf3d3b98a197d4d8aef46afce28170cae0808518ccd540ead5fc8 not found: ID does not exist" Sep 14 01:54:03.127009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.126990 2579 scope.go:117] "RemoveContainer" containerID="8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a" Sep 14 01:54:03.127277 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:54:03.127259 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a\": container with ID starting with 8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a not found: ID does not exist" containerID="8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a" Sep 14 01:54:03.127335 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.127286 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a"} err="failed to get container status \"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a\": rpc error: code = NotFound desc = could not find container \"8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a\": container with ID starting with 8024689a85053cb43aca65cc3d4af7dcc848e91a574a878ca48ca35479d7b13a not found: ID does not exist" Sep 14 01:54:03.131686 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.131663 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:54:03.135139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.135117 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-321811-openshift-default-7bfdf4974d-pz62m"] Sep 14 01:54:03.210923 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.210830 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" path="/var/lib/kubelet/pods/0cfba7ba-4d93-4b93-a7e0-c092f576ed80/volumes" Sep 14 01:54:03.553280 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:03.553238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:04.212172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.212119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:04.278997 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.278920 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:54:04.360070 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360025 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360122 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j54d\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360140 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360167 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360334 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:04.360517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360357 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360391 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket\") pod \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\" (UID: \"67561d02-e1e9-44f6-9c0a-7c33e8f71839\") " Sep 14 01:54:04.360724 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360575 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:04.360724 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360630 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:54:04.360833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360760 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data" (OuterVolumeSpecName: "istio-data") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:04.360833 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360770 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:04.360949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360882 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.360949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360904 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.360949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360918 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.360949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360932 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.360949 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.360945 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.362485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.362455 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d" (OuterVolumeSpecName: "kube-api-access-7j54d") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "kube-api-access-7j54d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:54:04.362624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.362484 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token" (OuterVolumeSpecName: "istio-token") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:54:04.362624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.362608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:54:04.363006 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.362985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67561d02-e1e9-44f6-9c0a-7c33e8f71839" (UID: "67561d02-e1e9-44f6-9c0a-7c33e8f71839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:54:04.420074 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.420017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:04.461735 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.461702 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.461735 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.461730 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j54d\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-kube-api-access-7j54d\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.461735 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.461741 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.461965 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.461749 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67561d02-e1e9-44f6-9c0a-7c33e8f71839-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:54:04.834588 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.834552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:04.935624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:04.935581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:05.126209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.126124 2579 generic.go:356] "Generic (PLEG): container finished" podID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" containerID="f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d" exitCode=0 Sep 14 01:54:05.126209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.126173 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" Sep 14 01:54:05.126209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.126189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" event={"ID":"67561d02-e1e9-44f6-9c0a-7c33e8f71839","Type":"ContainerDied","Data":"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d"} Sep 14 01:54:05.126427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.126274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb" event={"ID":"67561d02-e1e9-44f6-9c0a-7c33e8f71839","Type":"ContainerDied","Data":"1f2e198f32fbd49934fec42dfbcb1dbc0be40792ada904528bf92f6bcaa5c0e4"} Sep 14 01:54:05.126427 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.126304 2579 scope.go:117] "RemoveContainer" containerID="f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d" Sep 14 01:54:05.136920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.136768 2579 scope.go:117] "RemoveContainer" containerID="f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d" Sep 14 01:54:05.137135 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:54:05.137112 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d\": container with ID starting with f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d not found: ID does not exist" containerID="f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d" Sep 14 01:54:05.137188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.137143 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d"} err="failed to get container status \"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d\": rpc error: code = NotFound desc = could not find container \"f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d\": container with ID starting with f135a2127a034bce321b5e813d8c328f2e9cdbbce07d267c8d9d9b47d1d2624d not found: ID does not exist" Sep 14 01:54:05.147891 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.147865 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:54:05.151372 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.151350 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-06e7b2-openshift-default-5498d986d8-hdftb"] Sep 14 01:54:05.210368 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.210336 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" path="/var/lib/kubelet/pods/67561d02-e1e9-44f6-9c0a-7c33e8f71839/volumes" Sep 14 01:54:05.300625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.300587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:05.389297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.389215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:05.390946 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.390923 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 01:54:05.391379 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391362 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" containerName="istio-proxy" Sep 14 01:54:05.391379 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391378 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" containerName="istio-proxy" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391387 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391393 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391402 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5569b2-70f3-4958-80f4-a648b8640d8b" containerName="revoke-keys" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391407 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5569b2-70f3-4958-80f4-a648b8640d8b" containerName="revoke-keys" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391424 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b228d39c-ba77-4a57-9da4-39070f186b6f" containerName="payload-processing" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391430 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b228d39c-ba77-4a57-9da4-39070f186b6f" containerName="payload-processing" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391508 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b228d39c-ba77-4a57-9da4-39070f186b6f" containerName="payload-processing" Sep 14 01:54:05.391519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391517 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.391777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391524 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67561d02-e1e9-44f6-9c0a-7c33e8f71839" containerName="istio-proxy" Sep 14 01:54:05.391777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391533 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.391777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.391539 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5569b2-70f3-4958-80f4-a648b8640d8b" containerName="revoke-keys" Sep 14 01:54:05.395944 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.395928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.398536 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.398518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5d3ac7-dockercfg-hnhx6\"" Sep 14 01:54:05.398638 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.398572 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5d3ac7\"" Sep 14 01:54:05.404038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.404012 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 01:54:05.433168 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.433131 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 01:54:05.433633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.433611 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.433633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.433629 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cfba7ba-4d93-4b93-a7e0-c092f576ed80" containerName="istio-proxy" Sep 14 01:54:05.436845 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.436829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.446616 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.446590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 01:54:05.573721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.573691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.573918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.573750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtrs\" (UniqueName: \"kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.573918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.573826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.573918 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.573904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n67r8\" (UniqueName: \"kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.674982 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.674893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtrs\" (UniqueName: \"kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.674982 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.674948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.675199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.675000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n67r8\" (UniqueName: \"kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.675199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.675029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.675685 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.675663 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.675738 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.675719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.684228 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.684194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtrs\" (UniqueName: \"kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs\") pod \"payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.684384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.684317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n67r8\" (UniqueName: \"kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8\") pod \"payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.707938 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.707899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:05.749015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.748976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:05.983940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:05.983912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 01:54:05.985439 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:54:05.985408 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdaab2b9_e7ce_48bd_918e_683008ff22eb.slice/crio-ce73ecbc0348760e393072905fce8e3f207f899bce5449e3e544bf3f10d5b0d0 WatchSource:0}: Error finding container ce73ecbc0348760e393072905fce8e3f207f899bce5449e3e544bf3f10d5b0d0: Status 404 returned error can't find the container with id ce73ecbc0348760e393072905fce8e3f207f899bce5449e3e544bf3f10d5b0d0 Sep 14 01:54:06.042326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.042303 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 01:54:06.044369 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:54:06.044344 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1534c3a7_8dcc_4d2c_b18d_5c5bc3b9e73d.slice/crio-1025073c24c274d87af926f724bb37bd8ada5b506711fbf8ef608f1b8f73443c WatchSource:0}: Error finding container 1025073c24c274d87af926f724bb37bd8ada5b506711fbf8ef608f1b8f73443c: Status 404 returned error can't find the container with id 1025073c24c274d87af926f724bb37bd8ada5b506711fbf8ef608f1b8f73443c Sep 14 01:54:06.135895 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.135852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" event={"ID":"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d","Type":"ContainerStarted","Data":"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775"} Sep 14 01:54:06.136102 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.135923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:06.136102 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.135946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" event={"ID":"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d","Type":"ContainerStarted","Data":"1025073c24c274d87af926f724bb37bd8ada5b506711fbf8ef608f1b8f73443c"} Sep 14 01:54:06.137735 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.137704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" event={"ID":"bdaab2b9-e7ce-48bd-918e-683008ff22eb","Type":"ContainerStarted","Data":"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83"} Sep 14 01:54:06.137851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.137757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" event={"ID":"bdaab2b9-e7ce-48bd-918e-683008ff22eb","Type":"ContainerStarted","Data":"ce73ecbc0348760e393072905fce8e3f207f899bce5449e3e544bf3f10d5b0d0"} Sep 14 01:54:06.137851 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.137815 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:06.152870 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.152825 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" podStartSLOduration=1.152808851 podStartE2EDuration="1.152808851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:06.150495874 +0000 UTC m=+1391.383861156" watchObservedRunningTime="2026-09-14 01:54:06.152808851 +0000 UTC m=+1391.386174154" Sep 14 01:54:06.168188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:06.168117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" podStartSLOduration=1.168095944 podStartE2EDuration="1.168095944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:06.16434751 +0000 UTC m=+1391.397712793" watchObservedRunningTime="2026-09-14 01:54:06.168095944 +0000 UTC m=+1391.401461202" Sep 14 01:54:17.147019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:17.146988 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 01:54:17.147605 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:17.147089 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 01:54:27.643492 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:27.643448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:28.258351 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:28.258317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:28.849114 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:28.849076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:29.547074 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:29.547036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:30.150563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:30.150527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:30.926180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:30.926144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:31.393350 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:31.393311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:31.650207 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:31.650118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:31.750338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:31.750303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:31.999942 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:31.999905 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 01:54:32.004405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.004382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.006785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.006757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-86995c-openshift-default-dockercfg-p5s67\"" Sep 14 01:54:32.018485 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.018461 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 01:54:32.038921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.038874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:32.135220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135220 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135220 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjhq\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.135614 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.135591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.236710 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjhq\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.236905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.236905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.236905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.236905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.236979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.237006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237170 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.237031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237728 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.237700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.237951 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.237926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.238358 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.238337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.238943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.238660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.239100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.238868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.241010 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.240940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:32.242134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.242111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.242304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.242286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.252471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.252398 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.252637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.252615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjhq\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq\") pod \"e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.316426 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.316397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:32.595445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.595416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 01:54:32.596811 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:54:32.596780 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5019f7c7_e11e_42a5_94a0_9ada88d70fba.slice/crio-511daaaaa010442450f29657b3bfc8aa1899f6cadcd582219dba44b288d9cdfb WatchSource:0}: Error finding container 511daaaaa010442450f29657b3bfc8aa1899f6cadcd582219dba44b288d9cdfb: Status 404 returned error can't find the container with id 511daaaaa010442450f29657b3bfc8aa1899f6cadcd582219dba44b288d9cdfb Sep 14 01:54:32.599105 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.599045 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:54:32.599191 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.599136 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:54:32.599191 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:32.599167 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:54:33.300452 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:33.300414 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerStarted","Data":"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb"} Sep 14 01:54:33.300629 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:33.300468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerStarted","Data":"511daaaaa010442450f29657b3bfc8aa1899f6cadcd582219dba44b288d9cdfb"} Sep 14 01:54:33.316661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:33.316629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:33.318721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:33.318677 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podStartSLOduration=2.318658211 podStartE2EDuration="2.318658211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:33.318158435 +0000 UTC m=+1418.551523712" watchObservedRunningTime="2026-09-14 01:54:33.318658211 +0000 UTC m=+1418.552023468" Sep 14 01:54:33.322775 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:33.322749 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:34.307810 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:34.307773 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:34.308751 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:34.308730 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:54:34.633513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:34.633424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:35.185798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:35.185760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:35.286595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:35.286555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:35.401861 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:35.401832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:36.256544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:36.256511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:36.953190 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:36.953155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:37.660015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:37.659980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:38.264160 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:38.264124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:39.214157 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:39.214124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:39.541151 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:39.541116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:39.845813 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:39.845729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:39.945670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:39.945636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:40.054154 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.054114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:40.378920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.378877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:40.483086 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.483021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:40.484786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.484764 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 01:54:40.493024 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.492996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.496014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.495993 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-86995c\"" Sep 14 01:54:40.497082 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.497024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-86995c-dockercfg-ngxtt\"" Sep 14 01:54:40.499597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.499579 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 01:54:40.513595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.513574 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 01:54:40.517352 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.517335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.528983 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.528958 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 01:54:40.625514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.625470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlg2h\" (UniqueName: \"kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.626038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.625573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.626038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.625624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nxb\" (UniqueName: \"kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.626038 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.625690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.727178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlg2h\" (UniqueName: \"kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.727178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.727178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nxb\" (UniqueName: \"kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.727405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.727933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.727973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.727933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.734555 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.734524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlg2h\" (UniqueName: \"kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h\") pod \"payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:40.734677 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.734567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nxb\" (UniqueName: \"kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb\") pod \"payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.807447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.807409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:40.829653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:40.829623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:41.086794 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.086761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 01:54:41.089569 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:54:41.089541 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f0b4d7_d0ab_46e5_af46_dd5d1ba5ce93.slice/crio-b53c71026a3d67110361b737b00e7424d919d510d5b1526b9a52e2c4d5e6d778 WatchSource:0}: Error finding container b53c71026a3d67110361b737b00e7424d919d510d5b1526b9a52e2c4d5e6d778: Status 404 returned error can't find the container with id b53c71026a3d67110361b737b00e7424d919d510d5b1526b9a52e2c4d5e6d778 Sep 14 01:54:41.113029 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.113001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 01:54:41.113578 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:54:41.113554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec437c4f_9b5b_43e8_ae08_7352471debde.slice/crio-9fc882a1f0c0a39a9ae3a7b98e93ad9db9e1949f04e4fd6c7a4dbd2f52d3e804 WatchSource:0}: Error finding container 9fc882a1f0c0a39a9ae3a7b98e93ad9db9e1949f04e4fd6c7a4dbd2f52d3e804: Status 404 returned error can't find the container with id 9fc882a1f0c0a39a9ae3a7b98e93ad9db9e1949f04e4fd6c7a4dbd2f52d3e804 Sep 14 01:54:41.355099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.354982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" event={"ID":"ec437c4f-9b5b-43e8-ae08-7352471debde","Type":"ContainerStarted","Data":"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a"} Sep 14 01:54:41.355099 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.355068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" event={"ID":"ec437c4f-9b5b-43e8-ae08-7352471debde","Type":"ContainerStarted","Data":"9fc882a1f0c0a39a9ae3a7b98e93ad9db9e1949f04e4fd6c7a4dbd2f52d3e804"} Sep 14 01:54:41.355349 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.355206 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:41.356669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.356644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" event={"ID":"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93","Type":"ContainerStarted","Data":"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6"} Sep 14 01:54:41.356803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.356681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" event={"ID":"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93","Type":"ContainerStarted","Data":"b53c71026a3d67110361b737b00e7424d919d510d5b1526b9a52e2c4d5e6d778"} Sep 14 01:54:41.356803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.356737 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:41.371203 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.371146 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" podStartSLOduration=1.371133486 podStartE2EDuration="1.371133486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:41.369195306 +0000 UTC m=+1426.602560577" watchObservedRunningTime="2026-09-14 01:54:41.371133486 +0000 UTC m=+1426.604498737" Sep 14 01:54:41.386152 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.386101 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" podStartSLOduration=1.386086671 podStartE2EDuration="1.386086671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:41.38391351 +0000 UTC m=+1426.617278795" watchObservedRunningTime="2026-09-14 01:54:41.386086671 +0000 UTC m=+1426.619451928" Sep 14 01:54:41.729241 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:41.729207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:54:52.364326 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:52.364235 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 01:54:52.364844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:52.364683 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 01:54:55.415266 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:55.415235 2579 scope.go:117] "RemoveContainer" containerID="3fb3056aa37f2439459a6f88ef2031e1c3d229fd140ed381dce3db722bd2fc1f" Sep 14 01:54:55.423966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:54:55.423947 2579 scope.go:117] "RemoveContainer" containerID="2c7f84b95c6cd4be60c414443a5e425d6d2a7a3ff348968d3f77dd5ba91d30ac" Sep 14 01:55:00.222231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:00.222196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:01.775471 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:01.775435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:02.519140 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:02.519098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:03.133034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:03.132993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:03.744071 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:03.744015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:04.455754 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:04.455708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:04.883336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:04.883286 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": read tcp 10.134.0.2:56780->10.134.0.80:15021: read: connection reset by peer" start-of-body= Sep 14 01:55:04.883508 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:04.883381 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": read tcp 10.134.0.2:56780->10.134.0.80:15021: read: connection reset by peer" Sep 14 01:55:05.073645 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.073565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:05.245210 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.245174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:05.504651 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.504572 2579 generic.go:356] "Generic (PLEG): container finished" podID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerID="8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb" exitCode=0 Sep 14 01:55:05.505013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.504644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerDied","Data":"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb"} Sep 14 01:55:05.505090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.505035 2579 scope.go:117] "RemoveContainer" containerID="8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb" Sep 14 01:55:05.506431 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.506401 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:55:05.506514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.506487 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:55:05.506556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.506532 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:55:05.857628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:05.857523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:06.513348 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:06.513310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerStarted","Data":"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1"} Sep 14 01:55:06.552432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:06.552395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:07.317927 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:07.317879 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:55:07.318172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:07.318015 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 14 01:55:07.318172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:07.318076 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 14 01:55:08.317578 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:08.317531 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 14 01:55:08.318100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:08.317603 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 14 01:55:09.317468 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:09.317430 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 14 01:55:09.317788 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:09.317753 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 14 01:55:10.317544 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:10.317502 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 14 01:55:10.317765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:10.317590 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 14 01:55:11.317026 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:11.316983 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 14 01:55:11.317642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:11.317608 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 14 01:55:12.321330 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:12.321297 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:55:12.321782 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:12.321552 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:55:12.322188 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:12.322171 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 01:55:15.290811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.290779 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5"] Sep 14 01:55:15.293626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.293611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:15.296204 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.296184 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:55:15.296322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.296273 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:55:15.300808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.300784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5"] Sep 14 01:55:15.301279 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.301249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5\" (UID: \"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:15.402363 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.402322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5\" (UID: \"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:15.402980 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.402960 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5\" (UID: \"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:15.607487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.607395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:15.740393 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.740365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:15.879537 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:15.879499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5"] Sep 14 01:55:15.880927 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:55:15.880898 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc7f2b1_3d0b_4fef_aacf_b5bf726d23f5.slice/crio-26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b WatchSource:0}: Error finding container 26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b: Status 404 returned error can't find the container with id 26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b Sep 14 01:55:16.532029 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:16.531995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:16.584430 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:16.584397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" event={"ID":"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5","Type":"ContainerStarted","Data":"112dfa85d13fc93bc90f525b35b22413d549986316ec9bda6ef9218a3422e887"} Sep 14 01:55:16.584595 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:16.584453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" event={"ID":"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5","Type":"ContainerStarted","Data":"26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b"} Sep 14 01:55:16.602642 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:16.602552 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" podStartSLOduration=1.602525434 podStartE2EDuration="1.602525434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:55:16.599084801 +0000 UTC m=+1461.832450078" watchObservedRunningTime="2026-09-14 01:55:16.602525434 +0000 UTC m=+1461.835890693" Sep 14 01:55:17.215371 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:17.215331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:17.591900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:17.591866 2579 generic.go:356] "Generic (PLEG): container finished" podID="3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" containerID="112dfa85d13fc93bc90f525b35b22413d549986316ec9bda6ef9218a3422e887" exitCode=0 Sep 14 01:55:17.592339 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:17.591954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" event={"ID":"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5","Type":"ContainerDied","Data":"112dfa85d13fc93bc90f525b35b22413d549986316ec9bda6ef9218a3422e887"} Sep 14 01:55:17.935218 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:17.935135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:18.666801 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.666732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:18.735089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.734668 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:18.836825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.836792 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca\") pod \"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5\" (UID: \"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5\") " Sep 14 01:55:18.837148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.837127 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" (UID: "3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:55:18.848817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.848747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:18.938217 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:18.938190 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:55:19.253343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:19.253314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:19.604921 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:19.604877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" event={"ID":"3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5","Type":"ContainerDied","Data":"26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b"} Sep 14 01:55:19.605123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:19.604954 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a85c4b2915b2c7b02951c62e03b69ec2f625d430f6ac9048fb7515e0d2a95b" Sep 14 01:55:19.605123 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:19.604885 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5" Sep 14 01:55:20.006557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:20.006517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:20.711899 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:20.711851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:21.323543 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:21.323503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:30.849985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.849930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:30.873732 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.873533 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:55:30.874232 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.874193 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" podUID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" containerName="payload-processing" containerID="cri-o://7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336" gracePeriod=30 Sep 14 01:55:30.877765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.877732 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:55:30.878149 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.877919 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" podUID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" containerName="payload-pre-processing" containerID="cri-o://868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca" gracePeriod=30 Sep 14 01:55:30.917576 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.917536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:30.977569 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:30.977538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:31.020661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.020627 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5"] Sep 14 01:55:31.023195 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.023168 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-000f34a9-8rrq5"] Sep 14 01:55:31.036590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.036555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:31.190250 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.190222 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:55:31.193235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.193214 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:55:31.217807 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.217742 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" path="/var/lib/kubelet/pods/3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5/volumes" Sep 14 01:55:31.262556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262526 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume\") pod \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " Sep 14 01:55:31.262718 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume\") pod \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " Sep 14 01:55:31.262718 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262607 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdlhw\" (UniqueName: \"kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw\") pod \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\" (UID: \"17d8d736-8d1c-4f81-9d90-7591cc8263a2\") " Sep 14 01:55:31.262718 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262655 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7bs4\" (UniqueName: \"kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4\") pod \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\" (UID: \"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac\") " Sep 14 01:55:31.263470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17d8d736-8d1c-4f81-9d90-7591cc8263a2" (UID: "17d8d736-8d1c-4f81-9d90-7591cc8263a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:55:31.263470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.262977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" (UID: "8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:55:31.263470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.263196 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d8d736-8d1c-4f81-9d90-7591cc8263a2-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:55:31.263470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.263220 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:55:31.264778 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.264754 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw" (OuterVolumeSpecName: "kube-api-access-tdlhw") pod "17d8d736-8d1c-4f81-9d90-7591cc8263a2" (UID: "17d8d736-8d1c-4f81-9d90-7591cc8263a2"). InnerVolumeSpecName "kube-api-access-tdlhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:55:31.264879 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.264811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4" (OuterVolumeSpecName: "kube-api-access-f7bs4") pod "8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" (UID: "8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac"). InnerVolumeSpecName "kube-api-access-f7bs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:55:31.364334 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.364296 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlhw\" (UniqueName: \"kubernetes.io/projected/17d8d736-8d1c-4f81-9d90-7591cc8263a2-kube-api-access-tdlhw\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:55:31.364334 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.364328 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bs4\" (UniqueName: \"kubernetes.io/projected/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac-kube-api-access-f7bs4\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:55:31.525791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.525752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:31.676966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.676930 2579 generic.go:356] "Generic (PLEG): container finished" podID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" containerID="868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca" exitCode=0 Sep 14 01:55:31.677172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.676974 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" Sep 14 01:55:31.677172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.677015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" event={"ID":"17d8d736-8d1c-4f81-9d90-7591cc8263a2","Type":"ContainerDied","Data":"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca"} Sep 14 01:55:31.677172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.677078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k" event={"ID":"17d8d736-8d1c-4f81-9d90-7591cc8263a2","Type":"ContainerDied","Data":"374ca5dfc20d19321b42c8d04f9fbca645aca115ad9700ec110a3eeebed2f09a"} Sep 14 01:55:31.677172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.677109 2579 scope.go:117] "RemoveContainer" containerID="868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca" Sep 14 01:55:31.678505 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.678482 2579 generic.go:356] "Generic (PLEG): container finished" podID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" containerID="7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336" exitCode=0 Sep 14 01:55:31.678624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.678515 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" Sep 14 01:55:31.678624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.678537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" event={"ID":"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac","Type":"ContainerDied","Data":"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336"} Sep 14 01:55:31.678624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.678574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn" event={"ID":"8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac","Type":"ContainerDied","Data":"bf24cc81348b6b999a68a572eb5b41cb8159a8b0f414c1bcc539e918c25e6e11"} Sep 14 01:55:31.687720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.687263 2579 scope.go:117] "RemoveContainer" containerID="868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca" Sep 14 01:55:31.688636 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:55:31.688516 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca\": container with ID starting with 868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca not found: ID does not exist" containerID="868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca" Sep 14 01:55:31.688636 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.688547 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca"} err="failed to get container status \"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca\": rpc error: code = NotFound desc = could not find container \"868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca\": container with ID starting with 868850d1db7eafc5161d928e0e67a4762033798d33110d0e4ab1d4ff1e0800ca not found: ID does not exist" Sep 14 01:55:31.688636 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.688570 2579 scope.go:117] "RemoveContainer" containerID="7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336" Sep 14 01:55:31.697923 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.697906 2579 scope.go:117] "RemoveContainer" containerID="7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336" Sep 14 01:55:31.698208 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:55:31.698187 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336\": container with ID starting with 7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336 not found: ID does not exist" containerID="7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336" Sep 14 01:55:31.698256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.698218 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336"} err="failed to get container status \"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336\": rpc error: code = NotFound desc = could not find container \"7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336\": container with ID starting with 7389d25d8cb8fc3f7f041febb2189bb58b6a8633bbcc2b39e574d97526529336 not found: ID does not exist" Sep 14 01:55:31.705744 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.705715 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:55:31.709940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.709921 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-000f34a9-686c44d485-s2svn"] Sep 14 01:55:31.719355 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.719335 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:55:31.724772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:31.724748 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-000f34a9-5c9658cbf4-qt87k"] Sep 14 01:55:32.155189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:32.155148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:32.773582 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:32.773547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:33.211803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:33.211711 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" path="/var/lib/kubelet/pods/17d8d736-8d1c-4f81-9d90-7591cc8263a2/volumes" Sep 14 01:55:33.212315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:33.212287 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" path="/var/lib/kubelet/pods/8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac/volumes" Sep 14 01:55:33.446398 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:33.446369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:33.686274 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:33.686244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:33.786848 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:33.786818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:34.097283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:34.097253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:34.431660 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:34.431566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:34.835090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:34.835032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:35.445772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:35.445742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:39.454481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:39.454449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:42.539413 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:42.539378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:45.588399 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:45.588332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:55:55.276257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:55.276226 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:55:55.277883 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:55:55.277858 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 01:56:00.332016 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:00.331974 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:56:00.342689 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:00.342660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:08.308777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:08.308734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:15.351936 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.351883 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" podUID="41601062-844f-4398-9e74-ec159bb6523e" containerName="main" containerID="cri-o://7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd" gracePeriod=45 Sep 14 01:56:15.634262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.631816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:15.634262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.631945 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:56:15.712110 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.712043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:15.736819 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736892 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736929 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4txrv\" (UniqueName: \"kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737000 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.736974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm\") pod \"41601062-844f-4398-9e74-ec159bb6523e\" (UID: \"41601062-844f-4398-9e74-ec159bb6523e\") " Sep 14 01:56:15.737319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:15.737319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache" (OuterVolumeSpecName: "model-cache") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:15.737319 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home" (OuterVolumeSpecName: "home") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:15.737461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737383 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-tmp-dir\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.737461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737406 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-model-cache\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.737461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.737417 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-home\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.739337 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.739304 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm" (OuterVolumeSpecName: "dshm") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:15.739439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.739371 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv" (OuterVolumeSpecName: "kube-api-access-4txrv") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "kube-api-access-4txrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:56:15.739439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.739390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41601062-844f-4398-9e74-ec159bb6523e" (UID: "41601062-844f-4398-9e74-ec159bb6523e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:56:15.838356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.838310 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41601062-844f-4398-9e74-ec159bb6523e-tls-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.838356 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.838353 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txrv\" (UniqueName: \"kubernetes.io/projected/41601062-844f-4398-9e74-ec159bb6523e-kube-api-access-4txrv\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.838563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.838368 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41601062-844f-4398-9e74-ec159bb6523e-dshm\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:56:15.945206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.945121 2579 generic.go:356] "Generic (PLEG): container finished" podID="41601062-844f-4398-9e74-ec159bb6523e" containerID="7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd" exitCode=0 Sep 14 01:56:15.945206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.945179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" event={"ID":"41601062-844f-4398-9e74-ec159bb6523e","Type":"ContainerDied","Data":"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd"} Sep 14 01:56:15.945206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.945187 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" Sep 14 01:56:15.945206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.945210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv" event={"ID":"41601062-844f-4398-9e74-ec159bb6523e","Type":"ContainerDied","Data":"2c31fa4c5c0bb0c88b1469321a0ee11bf7074b0286feeec94216b6001ba2b2be"} Sep 14 01:56:15.945528 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.945227 2579 scope.go:117] "RemoveContainer" containerID="7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd" Sep 14 01:56:15.985179 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.985148 2579 scope.go:117] "RemoveContainer" containerID="7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd" Sep 14 01:56:15.985919 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:56:15.985879 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd\": container with ID starting with 7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd not found: ID does not exist" containerID="7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd" Sep 14 01:56:15.986177 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.986132 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd"} err="failed to get container status \"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd\": rpc error: code = NotFound desc = could not find container \"7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd\": container with ID starting with 7c0dd1883b186d3a79f934cb2ae97128874bf3adc0063d5df3228540aee943fd not found: ID does not exist" Sep 14 01:56:15.994088 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.993753 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:56:15.996417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:15.996377 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/e2e-conflict-a-9e04456b-kserve-64585887db-qv9pv"] Sep 14 01:56:16.610381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:16.610347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:17.211765 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:17.211732 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41601062-844f-4398-9e74-ec159bb6523e" path="/var/lib/kubelet/pods/41601062-844f-4398-9e74-ec159bb6523e/volumes" Sep 14 01:56:22.108786 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:22.108695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:27.731895 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:27.731856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:27.787542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:27.787506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:27.995788 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:27.995701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:29.199391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:29.199353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:29.299475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:29.299443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:29.399606 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:29.399559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:29.976314 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:29.976277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:30.076859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:30.076823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:30.179832 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:30.179800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:31.388144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:31.388105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:36.546432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:36.546387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:37.221964 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:37.221931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:37.323011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:37.322979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:37.883812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:37.883782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:43.412661 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:43.412631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:43.475355 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:43.475318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:43.557363 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:43.557325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:43.655351 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:43.655312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:46.935215 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:46.935174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:54.071299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:54.071267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:54.289773 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:54.289732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:55.480284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:55.480252 2579 scope.go:117] "RemoveContainer" containerID="d194d0421e064ec15c9bec54b37394752fa74f4233d5cc44289902d210a149f1" Sep 14 01:56:59.411803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.411765 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 01:56:59.412520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412499 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" containerName="revoke-keys" Sep 14 01:56:59.412520 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412521 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" containerName="revoke-keys" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412562 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41601062-844f-4398-9e74-ec159bb6523e" containerName="main" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412572 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41601062-844f-4398-9e74-ec159bb6523e" containerName="main" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412589 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" containerName="payload-pre-processing" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412599 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" containerName="payload-pre-processing" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412616 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" containerName="payload-processing" Sep 14 01:56:59.412704 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412624 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" containerName="payload-processing" Sep 14 01:56:59.413027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412754 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d8d736-8d1c-4f81-9d90-7591cc8263a2" containerName="payload-pre-processing" Sep 14 01:56:59.413027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412766 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc7f2b1-3d0b-4fef-aacf-b5bf726d23f5" containerName="revoke-keys" Sep 14 01:56:59.413027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412779 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f32d5a1-05e7-45e1-9c96-55cc8fd2f9ac" containerName="payload-processing" Sep 14 01:56:59.413027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.412790 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41601062-844f-4398-9e74-ec159bb6523e" containerName="main" Sep 14 01:56:59.416433 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.416416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.420256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.419982 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"kube-root-ca.crt\"" Sep 14 01:56:59.420256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.420165 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"default-dockercfg-j965m\"" Sep 14 01:56:59.420643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.420620 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"openshift-service-ca.crt\"" Sep 14 01:56:59.420771 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.420622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8e5935\"/\"distinct-2-w4-8e5935-kserve-self-signed-certs\"" Sep 14 01:56:59.424137 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.424113 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 01:56:59.428208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.428186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:56:59.468959 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.468927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.469113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.468978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnx7\" (UniqueName: \"kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.469113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.469006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.469113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.469033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.469246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.469138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.469246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.469209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnx7\" (UniqueName: \"kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570255 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570575 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570554 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570633 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.570669 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.570625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.572401 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.572381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.572693 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.572674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.577315 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.577291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnx7\" (UniqueName: \"kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7\") pod \"distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.726585 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.726501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:56:59.997074 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:56:59.997033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 01:57:00.002779 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:56:59.999744 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod823e58cd_d4e0_45f7_9751_e17b87cbb030.slice/crio-91840d1b5c92e32f6a2ffe77b9d09462ef7397df140e4992121e924bcad719c3 WatchSource:0}: Error finding container 91840d1b5c92e32f6a2ffe77b9d09462ef7397df140e4992121e924bcad719c3: Status 404 returned error can't find the container with id 91840d1b5c92e32f6a2ffe77b9d09462ef7397df140e4992121e924bcad719c3 Sep 14 01:57:00.203711 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:00.203671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" event={"ID":"823e58cd-d4e0-45f7-9751-e17b87cbb030","Type":"ContainerStarted","Data":"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd"} Sep 14 01:57:00.203864 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:00.203718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" event={"ID":"823e58cd-d4e0-45f7-9751-e17b87cbb030","Type":"ContainerStarted","Data":"91840d1b5c92e32f6a2ffe77b9d09462ef7397df140e4992121e924bcad719c3"} Sep 14 01:57:00.219391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:00.219330 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" podStartSLOduration=1.219309802 podStartE2EDuration="1.219309802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:00.218369095 +0000 UTC m=+1565.451734376" watchObservedRunningTime="2026-09-14 01:57:00.219309802 +0000 UTC m=+1565.452675062" Sep 14 01:57:09.726625 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:09.726586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:57:09.727021 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:09.726645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:57:09.739389 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:09.739364 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:57:10.272551 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:10.272522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 01:57:10.859835 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:10.859806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:10.960806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:10.960774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:11.343694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:11.343660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:15.270560 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:15.270524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:16.242698 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:16.242658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:20.785155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:20.785113 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:20.843779 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:20.843749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:21.146628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.146536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:21.562900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.562859 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:57:21.566835 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.566814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.569286 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.569267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b32209-openshift-default-dockercfg-gh5tm\"" Sep 14 01:57:21.581161 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.581131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:57:21.591791 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.591759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:21.648634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.648591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:21.693950 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.693902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.693989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh6q\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.694442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.694383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795447 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh6q\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.795925 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.796294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.796294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.795969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.796294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.796023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.796294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.796229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.796441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.796423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.798093 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.798071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.798264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.798246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.802793 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.802770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh6q\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.802877 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.802807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token\") pod \"e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:21.880103 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:21.880004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:22.155601 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.155569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:57:22.157431 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:57:22.157402 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf509abf3_ab18_44b8_aab1_7e677b59b0d2.slice/crio-17b292fa52ebd8b8fd5c7b80ccf502659057ab4a03898388533d27df4e72221b WatchSource:0}: Error finding container 17b292fa52ebd8b8fd5c7b80ccf502659057ab4a03898388533d27df4e72221b: Status 404 returned error can't find the container with id 17b292fa52ebd8b8fd5c7b80ccf502659057ab4a03898388533d27df4e72221b Sep 14 01:57:22.163256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.163220 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:22.163359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.163293 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:22.163359 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.163321 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:22.333800 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.333755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerStarted","Data":"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14"} Sep 14 01:57:22.334009 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.333819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerStarted","Data":"17b292fa52ebd8b8fd5c7b80ccf502659057ab4a03898388533d27df4e72221b"} Sep 14 01:57:22.352189 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.352119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podStartSLOduration=1.3521044039999999 podStartE2EDuration="1.352104404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:22.350588687 +0000 UTC m=+1587.583953969" watchObservedRunningTime="2026-09-14 01:57:22.352104404 +0000 UTC m=+1587.585469661" Sep 14 01:57:22.880524 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:22.880478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:23.612015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:23.611977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:23.886540 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:23.886456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:24.240140 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:24.240109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:24.342976 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:24.342928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:24.347115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:24.347038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:24.348239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:24.348215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:24.440734 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:24.440704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:25.137729 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:25.137700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:25.806470 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:25.806429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:26.625760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:26.625724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:27.235469 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:27.235412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:27.847514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:27.847481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:28.575593 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:28.575559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:28.675002 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:28.674971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:29.009375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:29.009337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:29.101266 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:29.101217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:50.430027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:50.429960 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:51.136869 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:51.136831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:51.842896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:51.842857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:52.458478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:52.458440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:53.070985 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.070955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:53.528845 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.528808 2579 generic.go:356] "Generic (PLEG): container finished" podID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerID="324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14" exitCode=0 Sep 14 01:57:53.529007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.528879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerDied","Data":"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14"} Sep 14 01:57:53.529351 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.529334 2579 scope.go:117] "RemoveContainer" containerID="324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14" Sep 14 01:57:53.530628 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.530602 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:53.530701 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.530673 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:53.530740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.530704 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:53.852261 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:53.852174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:54.346987 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.346953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:54.434785 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.434749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:54.543878 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.543832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:54.546933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.546899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerStarted","Data":"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52"} Sep 14 01:57:54.857414 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.857380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:54.880206 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.880178 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:54.881532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.881507 2579 patch_prober.go:28] interesting pod/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 14 01:57:54.881639 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:54.881555 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 14 01:57:55.075863 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.075828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:55.148866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.148784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:57:55.153483 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.153459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.157391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.157123 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-318a3c-openshift-default-dockercfg-m6rsv\"" Sep 14 01:57:55.158169 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158512 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158567 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpthc\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.158627 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.158620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.168549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.168525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:57:55.176468 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.176440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:55.259365 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.259796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.259785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpthc\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.260656 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.260454 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.260924 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.260881 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.261048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.261026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.261294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.261275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.261672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.261647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.264094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.264026 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.264094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.264070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.274420 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.274393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpthc\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.276937 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.276912 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token\") pod \"e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.375353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.375316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:55.466153 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.466049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:55.520086 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.506019 2579 scope.go:117] "RemoveContainer" containerID="a446650019c559af5ef34a3ef1718debd55b864ed4ccef039fd1d0c23a13bf77" Sep 14 01:57:55.771480 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.771454 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:57:55.773760 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:57:55.773733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005c04bf_6522_4ac7_aacb_5e42a423a091.slice/crio-9fd236d4c6c62146cf647572fb569deccd7e7e60addd318f9238eef21bba753e WatchSource:0}: Error finding container 9fd236d4c6c62146cf647572fb569deccd7e7e60addd318f9238eef21bba753e: Status 404 returned error can't find the container with id 9fd236d4c6c62146cf647572fb569deccd7e7e60addd318f9238eef21bba753e Sep 14 01:57:55.776581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.776553 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:55.776664 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.776625 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:55.776701 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.776677 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:57:55.880678 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.880642 2579 patch_prober.go:28] interesting pod/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 14 01:57:55.880839 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:55.880709 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 14 01:57:56.561019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:56.560985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerStarted","Data":"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c"} Sep 14 01:57:56.561453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:56.561038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerStarted","Data":"9fd236d4c6c62146cf647572fb569deccd7e7e60addd318f9238eef21bba753e"} Sep 14 01:57:56.584792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:56.584747 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podStartSLOduration=1.5847316519999999 podStartE2EDuration="1.584731652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:56.583669812 +0000 UTC m=+1621.817035092" watchObservedRunningTime="2026-09-14 01:57:56.584731652 +0000 UTC m=+1621.818096908" Sep 14 01:57:56.880665 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:56.880590 2579 patch_prober.go:28] interesting pod/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 14 01:57:56.880665 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:56.880645 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 14 01:57:57.467043 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.467006 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:57.472344 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.472318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:57.566971 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.566940 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:57.567827 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.567807 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:57:57.755186 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.755156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:57.881239 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.881200 2579 patch_prober.go:28] interesting pod/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 14 01:57:57.881417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:57.881257 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 14 01:57:58.519973 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:58.519930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:58.618400 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:58.618362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:58.724100 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:58.724042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:58.880903 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:58.880814 2579 patch_prober.go:28] interesting pod/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Sep 14 01:57:58.880903 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:58.880876 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Sep 14 01:57:59.428429 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:59.428390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:57:59.884740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:59.884710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:59.885148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:59.884972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:57:59.885743 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:57:59.885728 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:58:00.216082 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:00.215963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:00.839659 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:00.839607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:01.459299 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:01.459260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:02.084591 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:02.084555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:02.834670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:02.834633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:03.551784 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:03.551746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:03.603556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:03.603519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:03.983721 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:03.983690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:04.080417 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.080380 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:58:04.081956 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.081930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:04.084390 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.084372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.086684 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.086663 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-318a3c\"" Sep 14 01:58:04.086796 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.086755 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-318a3c-dockercfg-jwgcz\"" Sep 14 01:58:04.094350 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.094327 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:58:04.151558 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.151523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.151747 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.151670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.163717 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.163682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:04.253442 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.253357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.253792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.253765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.256994 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.256842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.263832 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.263800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k\") pod \"payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.397209 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.397171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:04.668790 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:04.668766 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:58:04.671701 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:58:04.671677 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc250dc0f_eb0f_40d3_9392_a031b6dcf072.slice/crio-98431d2436c3abad3849ad8365a6b40ad2bc4b5104824657700182575720fb15 WatchSource:0}: Error finding container 98431d2436c3abad3849ad8365a6b40ad2bc4b5104824657700182575720fb15: Status 404 returned error can't find the container with id 98431d2436c3abad3849ad8365a6b40ad2bc4b5104824657700182575720fb15 Sep 14 01:58:05.621131 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:05.621090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" event={"ID":"c250dc0f-eb0f-40d3-9392-a031b6dcf072","Type":"ContainerStarted","Data":"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac"} Sep 14 01:58:05.621515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:05.621162 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" event={"ID":"c250dc0f-eb0f-40d3-9392-a031b6dcf072","Type":"ContainerStarted","Data":"98431d2436c3abad3849ad8365a6b40ad2bc4b5104824657700182575720fb15"} Sep 14 01:58:05.621515 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:05.621190 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:05.637017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:05.636954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" podStartSLOduration=1.636933799 podStartE2EDuration="1.636933799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:58:05.635923815 +0000 UTC m=+1630.869289091" watchObservedRunningTime="2026-09-14 01:58:05.636933799 +0000 UTC m=+1630.870299056" Sep 14 01:58:16.178933 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:16.178895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:16.630133 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:16.630097 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:58:23.741332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.741297 2579 generic.go:356] "Generic (PLEG): container finished" podID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerID="f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c" exitCode=0 Sep 14 01:58:23.741853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.741377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerDied","Data":"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c"} Sep 14 01:58:23.741977 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.741959 2579 scope.go:117] "RemoveContainer" containerID="f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c" Sep 14 01:58:23.743242 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.743213 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:58:23.743340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.743278 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:58:23.743340 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:23.743310 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 01:58:24.749641 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:24.749608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerStarted","Data":"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e"} Sep 14 01:58:25.466433 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:25.466394 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:58:25.467608 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:25.467580 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 14 01:58:25.467730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:25.467631 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 14 01:58:25.739689 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:25.739655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:26.466428 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:26.466389 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 14 01:58:26.466822 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:26.466451 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 14 01:58:26.542702 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:26.542665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:27.304418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:27.304389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:27.466880 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:27.466842 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 14 01:58:27.467297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:27.466897 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 14 01:58:27.567505 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:27.567406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:58:27.948231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:27.948134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:28.467124 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:28.467093 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 14 01:58:28.467500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:28.467144 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 14 01:58:28.501446 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:28.501410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:28.740103 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:28.740035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:29.434302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:29.434269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:29.467236 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:29.467203 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 14 01:58:29.467611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:29.467271 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 14 01:58:30.212979 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:30.212923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:30.470812 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:30.470727 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:58:30.471760 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:30.471740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:58:30.836049 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:30.836016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:31.424225 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.424192 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9"] Sep 14 01:58:31.429213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.429188 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:31.433041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.433013 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:58:31.433397 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.433378 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:58:31.433609 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.433416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9"] Sep 14 01:58:31.447885 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.447851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:31.447885 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.447869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32209-gcmt9\" (UID: \"a09be4f3-bb86-4b55-a545-157716079ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:31.549437 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.549395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32209-gcmt9\" (UID: \"a09be4f3-bb86-4b55-a545-157716079ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:31.550199 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.550176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b32209-gcmt9\" (UID: \"a09be4f3-bb86-4b55-a545-157716079ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:31.744634 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:31.744602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:32.016615 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:32.016582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9"] Sep 14 01:58:32.018307 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:58:32.018277 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09be4f3_bb86_4b55_a545_157716079ea1.slice/crio-fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2 WatchSource:0}: Error finding container fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2: Status 404 returned error can't find the container with id fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2 Sep 14 01:58:32.336404 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:32.336298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:32.797601 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:32.797569 2579 generic.go:356] "Generic (PLEG): container finished" podID="a09be4f3-bb86-4b55-a545-157716079ea1" containerID="1f84751df2a62212a2750fb797466908b15fd4f734df8b8711c4b49e8fcf5578" exitCode=0 Sep 14 01:58:32.798022 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:32.797654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" event={"ID":"a09be4f3-bb86-4b55-a545-157716079ea1","Type":"ContainerDied","Data":"1f84751df2a62212a2750fb797466908b15fd4f734df8b8711c4b49e8fcf5578"} Sep 14 01:58:32.798022 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:32.797704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" event={"ID":"a09be4f3-bb86-4b55-a545-157716079ea1","Type":"ContainerStarted","Data":"fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2"} Sep 14 01:58:33.049095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:33.049023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:33.661998 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:33.661964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:33.946598 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:33.946573 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:33.977301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:33.977273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca\") pod \"a09be4f3-bb86-4b55-a545-157716079ea1\" (UID: \"a09be4f3-bb86-4b55-a545-157716079ea1\") " Sep 14 01:58:33.977603 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:33.977582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a09be4f3-bb86-4b55-a545-157716079ea1" (UID: "a09be4f3-bb86-4b55-a545-157716079ea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:34.078382 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.078346 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a09be4f3-bb86-4b55-a545-157716079ea1-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:34.352507 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.352469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:34.813590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.813556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" event={"ID":"a09be4f3-bb86-4b55-a545-157716079ea1","Type":"ContainerDied","Data":"fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2"} Sep 14 01:58:34.813590 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.813584 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9" Sep 14 01:58:34.813837 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.813605 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd741d9e9f48dfb06a86416faf03a939659e56dd7723b8c4448ef485403dbbf2" Sep 14 01:58:34.972256 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:34.972220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:35.756013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:35.755985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:36.366640 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:36.366595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:37.133080 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:37.133035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:37.856868 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:37.856830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:41.538380 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:41.538344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.153048 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.153011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.222155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.222031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.285994 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.285963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.331504 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.331470 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9"] Sep 14 01:58:47.333756 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.333734 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b32209-gcmt9"] Sep 14 01:58:47.345461 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.345433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.684892 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.684849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:47.833715 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:47.833678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:48.459653 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:48.459616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:49.212342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.212313 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09be4f3-bb86-4b55-a545-157716079ea1" path="/var/lib/kubelet/pods/a09be4f3-bb86-4b55-a545-157716079ea1/volumes" Sep 14 01:58:49.330523 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.330482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:49.528687 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.528649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:49.657235 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.657194 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb"] Sep 14 01:58:49.658391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.658363 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09be4f3-bb86-4b55-a545-157716079ea1" containerName="revoke-keys" Sep 14 01:58:49.658391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.658392 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09be4f3-bb86-4b55-a545-157716079ea1" containerName="revoke-keys" Sep 14 01:58:49.658650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.658635 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09be4f3-bb86-4b55-a545-157716079ea1" containerName="revoke-keys" Sep 14 01:58:49.663919 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.663895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:49.664626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.664603 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb"] Sep 14 01:58:49.666246 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.666224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:58:49.666422 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.666264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:58:49.738940 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.738907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb\" (UID: \"1d6e0fcf-2a1b-4765-ad95-b86acef0d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:49.840361 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.840266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb\" (UID: \"1d6e0fcf-2a1b-4765-ad95-b86acef0d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:49.840896 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.840872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb\" (UID: \"1d6e0fcf-2a1b-4765-ad95-b86acef0d783\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:49.975557 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:49.975515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:50.098777 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.098695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:50.246034 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.246004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb"] Sep 14 01:58:50.247398 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:58:50.247367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6e0fcf_2a1b_4765_ad95_b86acef0d783.slice/crio-e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67 WatchSource:0}: Error finding container e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67: Status 404 returned error can't find the container with id e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67 Sep 14 01:58:50.736549 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.736502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:50.913672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.913632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" event={"ID":"1d6e0fcf-2a1b-4765-ad95-b86acef0d783","Type":"ContainerStarted","Data":"1df63fa9d6018d3f828f6baac7aaba39f345c3fd8d876674ab3fce3a06d368a1"} Sep 14 01:58:50.913858 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.913697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" event={"ID":"1d6e0fcf-2a1b-4765-ad95-b86acef0d783","Type":"ContainerStarted","Data":"e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67"} Sep 14 01:58:50.929879 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:50.929828 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" podStartSLOduration=1.929812844 podStartE2EDuration="1.929812844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:58:50.927546368 +0000 UTC m=+1676.160911647" watchObservedRunningTime="2026-09-14 01:58:50.929812844 +0000 UTC m=+1676.163178098" Sep 14 01:58:51.445298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:51.445197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:51.922015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:51.921984 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d6e0fcf-2a1b-4765-ad95-b86acef0d783" containerID="1df63fa9d6018d3f828f6baac7aaba39f345c3fd8d876674ab3fce3a06d368a1" exitCode=0 Sep 14 01:58:51.922403 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:51.922083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" event={"ID":"1d6e0fcf-2a1b-4765-ad95-b86acef0d783","Type":"ContainerDied","Data":"1df63fa9d6018d3f828f6baac7aaba39f345c3fd8d876674ab3fce3a06d368a1"} Sep 14 01:58:52.249733 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:52.249701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:52.891596 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:52.891561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:53.103792 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.103765 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:53.175855 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.175758 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca\") pod \"1d6e0fcf-2a1b-4765-ad95-b86acef0d783\" (UID: \"1d6e0fcf-2a1b-4765-ad95-b86acef0d783\") " Sep 14 01:58:53.176148 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.176120 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d6e0fcf-2a1b-4765-ad95-b86acef0d783" (UID: "1d6e0fcf-2a1b-4765-ad95-b86acef0d783"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:53.176226 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.176218 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6e0fcf-2a1b-4765-ad95-b86acef0d783-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:53.722453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.722413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:53.834361 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.834327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:53.934652 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.934616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:53.936516 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.936483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" event={"ID":"1d6e0fcf-2a1b-4765-ad95-b86acef0d783","Type":"ContainerDied","Data":"e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67"} Sep 14 01:58:53.936635 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.936536 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8547cf18561fc837aa7c23f49eded445508d62e888fd6384ad6423a0c405e67" Sep 14 01:58:53.936635 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:53.936500 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb" Sep 14 01:58:54.061652 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.061619 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:58:54.061905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.061857 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" containerID="cri-o://9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52" gracePeriod=30 Sep 14 01:58:54.077467 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.077434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:54.386924 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.386844 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7"] Sep 14 01:58:54.387386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.387371 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6e0fcf-2a1b-4765-ad95-b86acef0d783" containerName="revoke-keys" Sep 14 01:58:54.387450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.387388 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6e0fcf-2a1b-4765-ad95-b86acef0d783" containerName="revoke-keys" Sep 14 01:58:54.387487 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.387469 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6e0fcf-2a1b-4765-ad95-b86acef0d783" containerName="revoke-keys" Sep 14 01:58:54.390746 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.390727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:54.392972 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.392951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:58:54.394458 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.394431 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:58:54.397538 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.397504 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7"] Sep 14 01:58:54.439995 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.439956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:54.489428 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.489399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7\" (UID: \"943388aa-85fb-49c5-bf4d-27011680d3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:54.590680 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.590644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7\" (UID: \"943388aa-85fb-49c5-bf4d-27011680d3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:54.591381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.591358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7\" (UID: \"943388aa-85fb-49c5-bf4d-27011680d3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:54.641304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.641204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:54.706113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.706046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:54.983144 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:54.983113 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7"] Sep 14 01:58:54.984682 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:58:54.984650 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943388aa_85fb_49c5_bf4d_27011680d3f8.slice/crio-f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c WatchSource:0}: Error finding container f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c: Status 404 returned error can't find the container with id f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c Sep 14 01:58:55.191712 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:55.191666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:55.951276 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:55.951242 2579 generic.go:356] "Generic (PLEG): container finished" podID="943388aa-85fb-49c5-bf4d-27011680d3f8" containerID="f0a9d6d08a6e2ecc5531b8a10e7949c9e293fb730da025b1065e7ab93e955203" exitCode=0 Sep 14 01:58:55.951684 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:55.951331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" event={"ID":"943388aa-85fb-49c5-bf4d-27011680d3f8","Type":"ContainerDied","Data":"f0a9d6d08a6e2ecc5531b8a10e7949c9e293fb730da025b1065e7ab93e955203"} Sep 14 01:58:55.951684 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:55.951431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" event={"ID":"943388aa-85fb-49c5-bf4d-27011680d3f8","Type":"ContainerStarted","Data":"f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c"} Sep 14 01:58:56.238304 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:56.238270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:56.956297 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:56.956252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:57.107096 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.107071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:57.226197 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.226118 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca\") pod \"943388aa-85fb-49c5-bf4d-27011680d3f8\" (UID: \"943388aa-85fb-49c5-bf4d-27011680d3f8\") " Sep 14 01:58:57.226469 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.226445 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "943388aa-85fb-49c5-bf4d-27011680d3f8" (UID: "943388aa-85fb-49c5-bf4d-27011680d3f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:57.327597 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.327563 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/943388aa-85fb-49c5-bf4d-27011680d3f8-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:57.623439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.623406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:57.966413 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.966335 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" Sep 14 01:58:57.966413 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.966370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7" event={"ID":"943388aa-85fb-49c5-bf4d-27011680d3f8","Type":"ContainerDied","Data":"f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c"} Sep 14 01:58:57.966807 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:57.966417 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2754022613a7b6fc484cebd2e89613affd88fc0022755a640cadb5fced3644c" Sep 14 01:58:58.446338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:58.446310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:59.050140 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.050107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:59.328500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.328478 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:58:59.451662 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451631 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.451853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451669 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.451853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451717 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.451853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451744 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phh6q\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.451853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451788 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.451853 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451827 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451857 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token\") pod \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\" (UID: \"f509abf3-ab18-44b8-aab1-7e677b59b0d2\") " Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.451978 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452108 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452218 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:58:59.452302 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452288 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:59.452708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452495 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.452708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452513 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.452708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452527 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.452708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452538 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.452708 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.452551 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.454026 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.454002 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:58:59.454262 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.454182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q" (OuterVolumeSpecName: "kube-api-access-phh6q") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "kube-api-access-phh6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:58:59.454457 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.454433 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:58:59.454542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.454455 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f509abf3-ab18-44b8-aab1-7e677b59b0d2" (UID: "f509abf3-ab18-44b8-aab1-7e677b59b0d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:58:59.553900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.553865 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.553900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.553892 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.553900 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.553909 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f509abf3-ab18-44b8-aab1-7e677b59b0d2-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.554147 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.553919 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh6q\" (UniqueName: \"kubernetes.io/projected/f509abf3-ab18-44b8-aab1-7e677b59b0d2-kube-api-access-phh6q\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:58:59.709869 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.709781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:58:59.988213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.988153 2579 generic.go:356] "Generic (PLEG): container finished" podID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerID="9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52" exitCode=0 Sep 14 01:58:59.988213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.988193 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerDied","Data":"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52"} Sep 14 01:58:59.988480 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.988223 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" Sep 14 01:58:59.988480 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.988256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8" event={"ID":"f509abf3-ab18-44b8-aab1-7e677b59b0d2","Type":"ContainerDied","Data":"17b292fa52ebd8b8fd5c7b80ccf502659057ab4a03898388533d27df4e72221b"} Sep 14 01:58:59.988480 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.988285 2579 scope.go:117] "RemoveContainer" containerID="9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52" Sep 14 01:58:59.998663 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:58:59.998646 2579 scope.go:117] "RemoveContainer" containerID="324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14" Sep 14 01:59:00.007874 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.007850 2579 scope.go:117] "RemoveContainer" containerID="9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52" Sep 14 01:59:00.008245 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:00.008222 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52\": container with ID starting with 9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52 not found: ID does not exist" containerID="9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52" Sep 14 01:59:00.008338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.008253 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52"} err="failed to get container status \"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52\": rpc error: code = NotFound desc = could not find container \"9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52\": container with ID starting with 9e1949ae2fb70c3259682234bb42a500d2a8fbf842ef6696872c56dc6f02be52 not found: ID does not exist" Sep 14 01:59:00.008338 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.008282 2579 scope.go:117] "RemoveContainer" containerID="324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14" Sep 14 01:59:00.008566 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:00.008550 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14\": container with ID starting with 324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14 not found: ID does not exist" containerID="324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14" Sep 14 01:59:00.008612 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.008571 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14"} err="failed to get container status \"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14\": rpc error: code = NotFound desc = could not find container \"324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14\": container with ID starting with 324e67c9c8ca4907448d2ef614fe9e6c8ff565053f9e614bd1aa50a53ac03b14 not found: ID does not exist" Sep 14 01:59:00.010134 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.010105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:59:00.013920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.013887 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b32209-openshift-default-66b75d7fd8-mz8c8"] Sep 14 01:59:00.446386 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:00.446285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:01.213397 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:01.213364 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" path="/var/lib/kubelet/pods/f509abf3-ab18-44b8-aab1-7e677b59b0d2/volumes" Sep 14 01:59:02.503564 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:02.503530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:02.554643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:02.554611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:04.533475 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:04.533434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.027322 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.027280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.054716 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.054681 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:59:06.054929 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.054901 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" podUID="a24f59e4-36ec-4d26-af2c-55cbadffa802" containerName="payload-processing" containerID="cri-o://3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c" gracePeriod=30 Sep 14 01:59:06.064747 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.063015 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:59:06.064747 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.063247 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" podUID="9eddaf66-7848-4f73-9ca0-1470091cae38" containerName="payload-pre-processing" containerID="cri-o://937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7" gracePeriod=30 Sep 14 01:59:06.095261 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.095224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.151952 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.151914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.207968 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.207894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb"] Sep 14 01:59:06.209802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.209625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.211798 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.211718 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-99c616-qk2qb"] Sep 14 01:59:06.399293 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.399269 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:59:06.401432 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.401414 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:59:06.426384 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426356 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbrcz\" (UniqueName: \"kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz\") pod \"a24f59e4-36ec-4d26-af2c-55cbadffa802\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " Sep 14 01:59:06.426556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume\") pod \"9eddaf66-7848-4f73-9ca0-1470091cae38\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " Sep 14 01:59:06.426556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume\") pod \"a24f59e4-36ec-4d26-af2c-55cbadffa802\" (UID: \"a24f59e4-36ec-4d26-af2c-55cbadffa802\") " Sep 14 01:59:06.426556 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlqgf\" (UniqueName: \"kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf\") pod \"9eddaf66-7848-4f73-9ca0-1470091cae38\" (UID: \"9eddaf66-7848-4f73-9ca0-1470091cae38\") " Sep 14 01:59:06.426820 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426792 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9eddaf66-7848-4f73-9ca0-1470091cae38" (UID: "9eddaf66-7848-4f73-9ca0-1470091cae38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:06.426897 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.426836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a24f59e4-36ec-4d26-af2c-55cbadffa802" (UID: "a24f59e4-36ec-4d26-af2c-55cbadffa802"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:06.429212 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.428729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf" (OuterVolumeSpecName: "kube-api-access-qlqgf") pod "9eddaf66-7848-4f73-9ca0-1470091cae38" (UID: "9eddaf66-7848-4f73-9ca0-1470091cae38"). InnerVolumeSpecName "kube-api-access-qlqgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:06.430872 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.429695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz" (OuterVolumeSpecName: "kube-api-access-fbrcz") pod "a24f59e4-36ec-4d26-af2c-55cbadffa802" (UID: "a24f59e4-36ec-4d26-af2c-55cbadffa802"). InnerVolumeSpecName "kube-api-access-fbrcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:06.528257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.528225 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrcz\" (UniqueName: \"kubernetes.io/projected/a24f59e4-36ec-4d26-af2c-55cbadffa802-kube-api-access-fbrcz\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.528257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.528251 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eddaf66-7848-4f73-9ca0-1470091cae38-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.528257 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.528262 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a24f59e4-36ec-4d26-af2c-55cbadffa802-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.528481 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.528272 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlqgf\" (UniqueName: \"kubernetes.io/projected/9eddaf66-7848-4f73-9ca0-1470091cae38-kube-api-access-qlqgf\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.582141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.582105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:06.685609 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:06.685453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:07.036017 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.035982 2579 generic.go:356] "Generic (PLEG): container finished" podID="9eddaf66-7848-4f73-9ca0-1470091cae38" containerID="937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7" exitCode=0 Sep 14 01:59:07.036500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.036026 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" Sep 14 01:59:07.036500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.036113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" event={"ID":"9eddaf66-7848-4f73-9ca0-1470091cae38","Type":"ContainerDied","Data":"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7"} Sep 14 01:59:07.036500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.036210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v" event={"ID":"9eddaf66-7848-4f73-9ca0-1470091cae38","Type":"ContainerDied","Data":"b36faf631b4d69c231eda4114ccd021d0dde2a4f5e8da91b1a04332a90315e46"} Sep 14 01:59:07.036500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.036243 2579 scope.go:117] "RemoveContainer" containerID="937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7" Sep 14 01:59:07.038113 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.038089 2579 generic.go:356] "Generic (PLEG): container finished" podID="a24f59e4-36ec-4d26-af2c-55cbadffa802" containerID="3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c" exitCode=0 Sep 14 01:59:07.038283 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.038254 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" Sep 14 01:59:07.038412 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.038257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" event={"ID":"a24f59e4-36ec-4d26-af2c-55cbadffa802","Type":"ContainerDied","Data":"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c"} Sep 14 01:59:07.038530 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.038445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp" event={"ID":"a24f59e4-36ec-4d26-af2c-55cbadffa802","Type":"ContainerDied","Data":"df6f1efacacc453aed8cbae53506319d89ef08fe51e9bc5c862e56f040ab5f6d"} Sep 14 01:59:07.047672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.047656 2579 scope.go:117] "RemoveContainer" containerID="937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7" Sep 14 01:59:07.047947 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:07.047929 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7\": container with ID starting with 937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7 not found: ID does not exist" containerID="937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7" Sep 14 01:59:07.048014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.047956 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7"} err="failed to get container status \"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7\": rpc error: code = NotFound desc = could not find container \"937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7\": container with ID starting with 937400b7b92be6ab662a908095b8821e27b7a1f51b1289212b38449c5de490c7 not found: ID does not exist" Sep 14 01:59:07.048014 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.047974 2579 scope.go:117] "RemoveContainer" containerID="3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c" Sep 14 01:59:07.056748 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.056560 2579 scope.go:117] "RemoveContainer" containerID="3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c" Sep 14 01:59:07.056848 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:07.056830 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c\": container with ID starting with 3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c not found: ID does not exist" containerID="3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c" Sep 14 01:59:07.056887 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.056856 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c"} err="failed to get container status \"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c\": rpc error: code = NotFound desc = could not find container \"3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c\": container with ID starting with 3a29df386471374e8d969e9d9b25bca18c15b3983eb4aeced72008f46d190a1c not found: ID does not exist" Sep 14 01:59:07.062039 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.062016 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:59:07.065500 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.065478 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-99c616-d9d7d7747-zwxsp"] Sep 14 01:59:07.077050 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.077015 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:59:07.080513 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.080490 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-99c616-c4bd86ff8-jk46v"] Sep 14 01:59:07.212932 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.212894 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6e0fcf-2a1b-4765-ad95-b86acef0d783" path="/var/lib/kubelet/pods/1d6e0fcf-2a1b-4765-ad95-b86acef0d783/volumes" Sep 14 01:59:07.213332 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.213317 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eddaf66-7848-4f73-9ca0-1470091cae38" path="/var/lib/kubelet/pods/9eddaf66-7848-4f73-9ca0-1470091cae38/volumes" Sep 14 01:59:07.213637 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.213625 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24f59e4-36ec-4d26-af2c-55cbadffa802" path="/var/lib/kubelet/pods/a24f59e4-36ec-4d26-af2c-55cbadffa802/volumes" Sep 14 01:59:07.342109 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.342010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:07.949803 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.949771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:07.985478 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.985444 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:59:07.985724 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:07.985693 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" podUID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" containerName="payload-pre-processing" containerID="cri-o://1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac" gracePeriod=30 Sep 14 01:59:08.012011 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.011980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:08.077538 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.077506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:08.176423 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.176320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:08.176876 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.176859 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7"] Sep 14 01:59:08.179307 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.179283 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-318a3c-7kfj7"] Sep 14 01:59:08.268842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.268815 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:59:08.277668 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.277638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:08.351277 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.351244 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k\") pod \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " Sep 14 01:59:08.351436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.351340 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume\") pod \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\" (UID: \"c250dc0f-eb0f-40d3-9392-a031b6dcf072\") " Sep 14 01:59:08.351694 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.351669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c250dc0f-eb0f-40d3-9392-a031b6dcf072" (UID: "c250dc0f-eb0f-40d3-9392-a031b6dcf072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:08.353375 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.353350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k" (OuterVolumeSpecName: "kube-api-access-k529k") pod "c250dc0f-eb0f-40d3-9392-a031b6dcf072" (UID: "c250dc0f-eb0f-40d3-9392-a031b6dcf072"). InnerVolumeSpecName "kube-api-access-k529k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:08.452889 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.452859 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k529k\" (UniqueName: \"kubernetes.io/projected/c250dc0f-eb0f-40d3-9392-a031b6dcf072-kube-api-access-k529k\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:08.452889 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.452883 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c250dc0f-eb0f-40d3-9392-a031b6dcf072-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:08.532905 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.532820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:08.983518 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:08.983491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:09.058692 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.058655 2579 generic.go:356] "Generic (PLEG): container finished" podID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" containerID="1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac" exitCode=0 Sep 14 01:59:09.058862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.058712 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" Sep 14 01:59:09.058862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.058708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" event={"ID":"c250dc0f-eb0f-40d3-9392-a031b6dcf072","Type":"ContainerDied","Data":"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac"} Sep 14 01:59:09.058862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.058833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh" event={"ID":"c250dc0f-eb0f-40d3-9392-a031b6dcf072","Type":"ContainerDied","Data":"98431d2436c3abad3849ad8365a6b40ad2bc4b5104824657700182575720fb15"} Sep 14 01:59:09.058862 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.058858 2579 scope.go:117] "RemoveContainer" containerID="1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac" Sep 14 01:59:09.068831 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.068811 2579 scope.go:117] "RemoveContainer" containerID="1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac" Sep 14 01:59:09.069110 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:09.069091 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac\": container with ID starting with 1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac not found: ID does not exist" containerID="1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac" Sep 14 01:59:09.069182 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.069116 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac"} err="failed to get container status \"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac\": rpc error: code = NotFound desc = could not find container \"1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac\": container with ID starting with 1baea3b5ddb81481e131cd88caff13c4f252860f541ed32b4b37a35ce7ed28ac not found: ID does not exist" Sep 14 01:59:09.079316 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.079291 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:59:09.090826 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.090806 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-318a3c-54686f84db-xwcjh"] Sep 14 01:59:09.211142 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.211097 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="943388aa-85fb-49c5-bf4d-27011680d3f8" path="/var/lib/kubelet/pods/943388aa-85fb-49c5-bf4d-27011680d3f8/volumes" Sep 14 01:59:09.211581 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.211561 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" path="/var/lib/kubelet/pods/c250dc0f-eb0f-40d3-9392-a031b6dcf072/volumes" Sep 14 01:59:09.724826 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:09.724781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:10.341139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:10.341105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:10.946436 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:10.946399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:11.655808 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:11.655777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:12.256181 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.256153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:12.708667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.708551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:12.808770 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.808731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:12.955802 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.955767 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:59:12.956090 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.956036 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" containerID="cri-o://e22af9638cf91eb536f25090c76b28c90dde716cdde3c1604b3cd00d00b93490" gracePeriod=30 Sep 14 01:59:12.966779 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:12.966699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:14.278441 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.278411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:14.335818 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.335782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:14.527117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.521047 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:59:14.529141 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.528729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:14.529804 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.529341 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" containerID="cri-o://43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e" gracePeriod=30 Sep 14 01:59:14.849342 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849250 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g"] Sep 14 01:59:14.849859 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849838 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849861 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849872 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849880 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849908 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943388aa-85fb-49c5-bf4d-27011680d3f8" containerName="revoke-keys" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849917 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="943388aa-85fb-49c5-bf4d-27011680d3f8" containerName="revoke-keys" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849931 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" containerName="payload-pre-processing" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849939 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" containerName="payload-pre-processing" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849961 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24f59e4-36ec-4d26-af2c-55cbadffa802" containerName="payload-processing" Sep 14 01:59:14.849966 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849969 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24f59e4-36ec-4d26-af2c-55cbadffa802" containerName="payload-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849982 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eddaf66-7848-4f73-9ca0-1470091cae38" containerName="payload-pre-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.849990 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eddaf66-7848-4f73-9ca0-1470091cae38" containerName="payload-pre-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.850124 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.850138 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c250dc0f-eb0f-40d3-9392-a031b6dcf072" containerName="payload-pre-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.850147 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24f59e4-36ec-4d26-af2c-55cbadffa802" containerName="payload-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.850159 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eddaf66-7848-4f73-9ca0-1470091cae38" containerName="payload-pre-processing" Sep 14 01:59:14.850418 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.850170 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="943388aa-85fb-49c5-bf4d-27011680d3f8" containerName="revoke-keys" Sep 14 01:59:14.855041 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.855020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:14.857354 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.857331 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:59:14.857506 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.857409 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:59:14.859602 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.859573 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g"] Sep 14 01:59:14.925155 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:14.925128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g\" (UID: \"dd0da4f0-4ef9-4170-a4d9-c915d0afe111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:15.014003 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.013963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:15.026270 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.026243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g\" (UID: \"dd0da4f0-4ef9-4170-a4d9-c915d0afe111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:15.026868 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.026851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g\" (UID: \"dd0da4f0-4ef9-4170-a4d9-c915d0afe111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:15.169287 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.169201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:15.445517 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.445487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g"] Sep 14 01:59:15.446926 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:59:15.446900 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0da4f0_4ef9_4170_a4d9_c915d0afe111.slice/crio-2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323 WatchSource:0}: Error finding container 2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323: Status 404 returned error can't find the container with id 2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323 Sep 14 01:59:15.474030 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.474000 2579 patch_prober.go:28] interesting pod/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:59:15.474157 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.474054 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:59:15.839781 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:15.839742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:16.111405 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:16.111370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" event={"ID":"dd0da4f0-4ef9-4170-a4d9-c915d0afe111","Type":"ContainerStarted","Data":"cef302248dfed8e54d24707950d740a98e99cd5bd88574842388af3951f4d855"} Sep 14 01:59:16.111542 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:16.111440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" event={"ID":"dd0da4f0-4ef9-4170-a4d9-c915d0afe111","Type":"ContainerStarted","Data":"2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323"} Sep 14 01:59:16.627381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:16.627345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:17.120611 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:17.120575 2579 generic.go:356] "Generic (PLEG): container finished" podID="dd0da4f0-4ef9-4170-a4d9-c915d0afe111" containerID="cef302248dfed8e54d24707950d740a98e99cd5bd88574842388af3951f4d855" exitCode=0 Sep 14 01:59:17.120805 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:17.120659 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" event={"ID":"dd0da4f0-4ef9-4170-a4d9-c915d0afe111","Type":"ContainerDied","Data":"cef302248dfed8e54d24707950d740a98e99cd5bd88574842388af3951f4d855"} Sep 14 01:59:17.267772 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:17.267736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:17.929609 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:17.929528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:18.131301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.131272 2579 generic.go:356] "Generic (PLEG): container finished" podID="a23fc97e-d029-416a-a4f9-e1789b960950" containerID="e22af9638cf91eb536f25090c76b28c90dde716cdde3c1604b3cd00d00b93490" exitCode=0 Sep 14 01:59:18.131438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.131349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerDied","Data":"e22af9638cf91eb536f25090c76b28c90dde716cdde3c1604b3cd00d00b93490"} Sep 14 01:59:18.131438 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.131424 2579 scope.go:117] "RemoveContainer" containerID="3493a436bc1c94678c2f00bf0bb9d88768e6ca812db17b36fed2801140304ae7" Sep 14 01:59:18.234866 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.234840 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:59:18.287656 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.287630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:18.359626 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.359797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.359797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359665 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.359797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359682 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.359797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.359797 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359784 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca\") pod \"dd0da4f0-4ef9-4170-a4d9-c915d0afe111\" (UID: \"dd0da4f0-4ef9-4170-a4d9-c915d0afe111\") " Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359915 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359950 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2zh7\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.359973 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:18.360095 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360053 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data\") pod \"a23fc97e-d029-416a-a4f9-e1789b960950\" (UID: \"a23fc97e-d029-416a-a4f9-e1789b960950\") " Sep 14 01:59:18.360407 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360123 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:18.360407 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360234 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:18.360407 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360290 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd0da4f0-4ef9-4170-a4d9-c915d0afe111" (UID: "dd0da4f0-4ef9-4170-a4d9-c915d0afe111"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:18.360407 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data" (OuterVolumeSpecName: "istio-data") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:18.360615 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360603 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360620 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360630 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360639 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360647 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23fc97e-d029-416a-a4f9-e1789b960950-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.360656 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0da4f0-4ef9-4170-a4d9-c915d0afe111-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.362047 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.362026 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:59:18.362172 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.362126 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7" (OuterVolumeSpecName: "kube-api-access-q2zh7") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "kube-api-access-q2zh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:18.362285 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.362266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:18.362439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.362426 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token" (OuterVolumeSpecName: "istio-token") pod "a23fc97e-d029-416a-a4f9-e1789b960950" (UID: "a23fc97e-d029-416a-a4f9-e1789b960950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:18.461275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.461192 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23fc97e-d029-416a-a4f9-e1789b960950-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.461275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.461223 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zh7\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-kube-api-access-q2zh7\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.461275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.461234 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23fc97e-d029-416a-a4f9-e1789b960950-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.461275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.461243 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23fc97e-d029-416a-a4f9-e1789b960950-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:18.698243 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:18.698190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:19.139284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.139253 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" Sep 14 01:59:19.139683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.139284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g" event={"ID":"dd0da4f0-4ef9-4170-a4d9-c915d0afe111","Type":"ContainerDied","Data":"2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323"} Sep 14 01:59:19.139683 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.139338 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2966ae92b80b344609c4796bf43a0d842431f04768f33120979f9e76c714e323" Sep 14 01:59:19.140706 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.140680 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" Sep 14 01:59:19.140811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.140718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4" event={"ID":"a23fc97e-d029-416a-a4f9-e1789b960950","Type":"ContainerDied","Data":"17ae18275fc8203f1f125ed15a1f5750794ae06f8d96a81f9c50d42fe3ca2569"} Sep 14 01:59:19.140811 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.140775 2579 scope.go:117] "RemoveContainer" containerID="e22af9638cf91eb536f25090c76b28c90dde716cdde3c1604b3cd00d00b93490" Sep 14 01:59:19.168226 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.168201 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:59:19.174027 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.173997 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-99c616-openshift-default-54d44b8c68-57zj4"] Sep 14 01:59:19.215589 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.215553 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" path="/var/lib/kubelet/pods/a23fc97e-d029-416a-a4f9-e1789b960950/volumes" Sep 14 01:59:19.333382 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.333288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:19.794391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.794366 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:59:19.874570 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874643 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874740 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874718 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874806 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpthc\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.874961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874945 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data\") pod \"005c04bf-6522-4ac7-aacb-5e42a423a091\" (UID: \"005c04bf-6522-4ac7-aacb-5e42a423a091\") " Sep 14 01:59:19.875208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874958 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:19.875208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.874973 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:19.875208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875053 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:19.875208 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875088 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:19.875391 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875268 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data" (OuterVolumeSpecName: "istio-data") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:19.875445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875413 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.875445 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875434 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.875519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875450 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.875519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875463 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005c04bf-6522-4ac7-aacb-5e42a423a091-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.875519 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.875489 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.876954 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.876926 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:19.877081 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.877046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc" (OuterVolumeSpecName: "kube-api-access-rpthc") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "kube-api-access-rpthc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:19.877277 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.877257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token" (OuterVolumeSpecName: "istio-token") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:19.877367 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.877348 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "005c04bf-6522-4ac7-aacb-5e42a423a091" (UID: "005c04bf-6522-4ac7-aacb-5e42a423a091"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:59:19.977007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.976931 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.977007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.976961 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.977007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.976970 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005c04bf-6522-4ac7-aacb-5e42a423a091-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:19.977007 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:19.976978 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpthc\" (UniqueName: \"kubernetes.io/projected/005c04bf-6522-4ac7-aacb-5e42a423a091-kube-api-access-rpthc\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:20.148910 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.148875 2579 generic.go:356] "Generic (PLEG): container finished" podID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerID="43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e" exitCode=0 Sep 14 01:59:20.149341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.148920 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" Sep 14 01:59:20.149341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.148959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerDied","Data":"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e"} Sep 14 01:59:20.149341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.149016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2" event={"ID":"005c04bf-6522-4ac7-aacb-5e42a423a091","Type":"ContainerDied","Data":"9fd236d4c6c62146cf647572fb569deccd7e7e60addd318f9238eef21bba753e"} Sep 14 01:59:20.149341 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.149045 2579 scope.go:117] "RemoveContainer" containerID="43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e" Sep 14 01:59:20.159300 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.159281 2579 scope.go:117] "RemoveContainer" containerID="f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c" Sep 14 01:59:20.168284 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.168253 2579 scope.go:117] "RemoveContainer" containerID="43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e" Sep 14 01:59:20.168609 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:20.168584 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e\": container with ID starting with 43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e not found: ID does not exist" containerID="43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e" Sep 14 01:59:20.168720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.168618 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e"} err="failed to get container status \"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e\": rpc error: code = NotFound desc = could not find container \"43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e\": container with ID starting with 43491dd5a416df96309e8c52938549021a061b5a44d1f8c27a6fd625df488e9e not found: ID does not exist" Sep 14 01:59:20.168720 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.168661 2579 scope.go:117] "RemoveContainer" containerID="f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c" Sep 14 01:59:20.172511 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:20.170336 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c\": container with ID starting with f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c not found: ID does not exist" containerID="f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c" Sep 14 01:59:20.172511 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.170368 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c"} err="failed to get container status \"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c\": rpc error: code = NotFound desc = could not find container \"f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c\": container with ID starting with f76f582ff4c06ae64c80c93610d152bf1d357ac4062c47dbb044345bcda5c53c not found: ID does not exist" Sep 14 01:59:20.174180 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.174159 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:59:20.176294 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:20.176275 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-318a3c-openshift-default-57cdd5c488-6nzg2"] Sep 14 01:59:21.211514 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:21.211482 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" path="/var/lib/kubelet/pods/005c04bf-6522-4ac7-aacb-5e42a423a091/volumes" Sep 14 01:59:25.330390 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.330356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:25.350506 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.350472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:59:25.430275 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.430243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:25.495532 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.495499 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g"] Sep 14 01:59:25.497015 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.496988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:25.498157 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.498139 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-38e192-57l4g"] Sep 14 01:59:25.616264 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.616173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:25.860454 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.860422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:25.960681 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:25.960586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:26.185545 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.185504 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" podUID="9affd72e-8204-4929-954e-5918db2cebe1" containerName="payload-processing" containerID="cri-o://e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e" gracePeriod=30 Sep 14 01:59:26.442446 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.442422 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:59:26.539914 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.539880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c746p\" (UniqueName: \"kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p\") pod \"9affd72e-8204-4929-954e-5918db2cebe1\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " Sep 14 01:59:26.540112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.539924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume\") pod \"9affd72e-8204-4929-954e-5918db2cebe1\" (UID: \"9affd72e-8204-4929-954e-5918db2cebe1\") " Sep 14 01:59:26.540402 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.540380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9affd72e-8204-4929-954e-5918db2cebe1" (UID: "9affd72e-8204-4929-954e-5918db2cebe1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:26.541955 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.541930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p" (OuterVolumeSpecName: "kube-api-access-c746p") pod "9affd72e-8204-4929-954e-5918db2cebe1" (UID: "9affd72e-8204-4929-954e-5918db2cebe1"). InnerVolumeSpecName "kube-api-access-c746p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:26.641115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.640997 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c746p\" (UniqueName: \"kubernetes.io/projected/9affd72e-8204-4929-954e-5918db2cebe1-kube-api-access-c746p\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:26.641115 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.641024 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9affd72e-8204-4929-954e-5918db2cebe1-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:26.763643 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:26.763611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:27.192117 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.192052 2579 generic.go:356] "Generic (PLEG): container finished" podID="9affd72e-8204-4929-954e-5918db2cebe1" containerID="e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e" exitCode=0 Sep 14 01:59:27.192282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.192123 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" Sep 14 01:59:27.192282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.192094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" event={"ID":"9affd72e-8204-4929-954e-5918db2cebe1","Type":"ContainerDied","Data":"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e"} Sep 14 01:59:27.192282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.192213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs" event={"ID":"9affd72e-8204-4929-954e-5918db2cebe1","Type":"ContainerDied","Data":"d650214bd5f84408716b1c9b5c58aabd1e8191b00bab9cd1658606de1b3368a5"} Sep 14 01:59:27.192282 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.192256 2579 scope.go:117] "RemoveContainer" containerID="e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e" Sep 14 01:59:27.202484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.202461 2579 scope.go:117] "RemoveContainer" containerID="e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e" Sep 14 01:59:27.202749 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:27.202728 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e\": container with ID starting with e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e not found: ID does not exist" containerID="e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e" Sep 14 01:59:27.202806 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.202758 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e"} err="failed to get container status \"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e\": rpc error: code = NotFound desc = could not find container \"e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e\": container with ID starting with e0515901fc91260f6857f98c2c89267532d1cf2a9eabf0e66584674364c80a8e not found: ID does not exist" Sep 14 01:59:27.210357 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.210328 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd0da4f0-4ef9-4170-a4d9-c915d0afe111" path="/var/lib/kubelet/pods/dd0da4f0-4ef9-4170-a4d9-c915d0afe111/volumes" Sep 14 01:59:27.214334 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.214307 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:59:27.216290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.216271 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-38e192-6467d6676f-jb9gs"] Sep 14 01:59:27.533484 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:27.533448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:28.259329 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:28.259288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:29.041943 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:29.041905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:29.211271 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:29.211232 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9affd72e-8204-4929-954e-5918db2cebe1" path="/var/lib/kubelet/pods/9affd72e-8204-4929-954e-5918db2cebe1/volumes" Sep 14 01:59:29.661374 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:29.661344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.374932 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.374893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.436439 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.436403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.529213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.529173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.599490 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.599434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.658176 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.658087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.731922 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.731885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:32.942453 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942378 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt"] Sep 14 01:59:32.942817 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942803 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942830 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9affd72e-8204-4929-954e-5918db2cebe1" containerName="payload-processing" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942835 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9affd72e-8204-4929-954e-5918db2cebe1" containerName="payload-processing" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942843 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0da4f0-4ef9-4170-a4d9-c915d0afe111" containerName="revoke-keys" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942849 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0da4f0-4ef9-4170-a4d9-c915d0afe111" containerName="revoke-keys" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942857 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942862 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.942867 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942867 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942874 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942882 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942887 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942945 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942954 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9affd72e-8204-4929-954e-5918db2cebe1" containerName="payload-processing" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942959 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23fc97e-d029-416a-a4f9-e1789b960950" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942965 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f509abf3-ab18-44b8-aab1-7e677b59b0d2" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942971 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942978 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="005c04bf-6522-4ac7-aacb-5e42a423a091" containerName="istio-proxy" Sep 14 01:59:32.943139 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.942984 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0da4f0-4ef9-4170-a4d9-c915d0afe111" containerName="revoke-keys" Sep 14 01:59:32.946089 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.946049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:32.948624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.948429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:59:32.948624 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.948436 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:59:32.952213 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:32.952193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt"] Sep 14 01:59:33.103961 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.103921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt\" (UID: \"ffc40204-15d4-40f3-a565-3fdb48f82ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:33.204550 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.204463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt\" (UID: \"ffc40204-15d4-40f3-a565-3fdb48f82ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:33.205019 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.204999 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt\" (UID: \"ffc40204-15d4-40f3-a565-3fdb48f82ffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:33.234809 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.234767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:33.257567 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.257529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:33.535670 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:33.535642 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt"] Sep 14 01:59:33.537554 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:59:33.537525 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc40204_15d4_40f3_a565_3fdb48f82ffd.slice/crio-dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044 WatchSource:0}: Error finding container dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044: Status 404 returned error can't find the container with id dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044 Sep 14 01:59:34.027163 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:34.027124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:34.236594 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:34.236565 2579 generic.go:356] "Generic (PLEG): container finished" podID="ffc40204-15d4-40f3-a565-3fdb48f82ffd" containerID="ae6bbd7a793994af14018b32731a41b583d07c7ef743b66cdcd97dda6857334d" exitCode=0 Sep 14 01:59:34.236730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:34.236642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" event={"ID":"ffc40204-15d4-40f3-a565-3fdb48f82ffd","Type":"ContainerDied","Data":"ae6bbd7a793994af14018b32731a41b583d07c7ef743b66cdcd97dda6857334d"} Sep 14 01:59:34.236730 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:34.236683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" event={"ID":"ffc40204-15d4-40f3-a565-3fdb48f82ffd","Type":"ContainerStarted","Data":"dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044"} Sep 14 01:59:34.662269 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:34.662239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:35.388178 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:35.388154 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:35.412774 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:35.412706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:35.528102 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:35.527995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca\") pod \"ffc40204-15d4-40f3-a565-3fdb48f82ffd\" (UID: \"ffc40204-15d4-40f3-a565-3fdb48f82ffd\") " Sep 14 01:59:35.528381 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:35.528360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffc40204-15d4-40f3-a565-3fdb48f82ffd" (UID: "ffc40204-15d4-40f3-a565-3fdb48f82ffd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:35.629667 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:35.629627 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffc40204-15d4-40f3-a565-3fdb48f82ffd-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:36.030857 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:36.030792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:36.250353 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:36.250319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" event={"ID":"ffc40204-15d4-40f3-a565-3fdb48f82ffd","Type":"ContainerDied","Data":"dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044"} Sep 14 01:59:36.250495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:36.250368 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd74ad7d8dee9aedb50599ff2d280dc7ade10971298c782f5c9bb37476f1d044" Sep 14 01:59:36.250495 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:36.250331 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt" Sep 14 01:59:36.637873 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:36.637832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:38.293111 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:38.293054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:43.350370 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.350333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:43.413035 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.412999 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:59:43.413301 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.413268 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" podUID="22c76e55-4873-4a58-9ba5-c9cffedc1798" containerName="payload-pre-processing" containerID="cri-o://3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f" gracePeriod=30 Sep 14 01:59:43.440613 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.440583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:43.563869 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.562483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:43.657219 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.657140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt"] Sep 14 01:59:43.662112 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.662088 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7ff93e-fbnqt"] Sep 14 01:59:43.676609 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.676579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:43.681999 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.681978 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:59:43.709960 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.709934 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume\") pod \"22c76e55-4873-4a58-9ba5-c9cffedc1798\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " Sep 14 01:59:43.710128 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.709974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b9jx\" (UniqueName: \"kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx\") pod \"22c76e55-4873-4a58-9ba5-c9cffedc1798\" (UID: \"22c76e55-4873-4a58-9ba5-c9cffedc1798\") " Sep 14 01:59:43.710672 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.710640 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22c76e55-4873-4a58-9ba5-c9cffedc1798" (UID: "22c76e55-4873-4a58-9ba5-c9cffedc1798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:43.712494 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.712464 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx" (OuterVolumeSpecName: "kube-api-access-4b9jx") pod "22c76e55-4873-4a58-9ba5-c9cffedc1798" (UID: "22c76e55-4873-4a58-9ba5-c9cffedc1798"). InnerVolumeSpecName "kube-api-access-4b9jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:43.811450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.811410 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c76e55-4873-4a58-9ba5-c9cffedc1798-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:43.811450 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.811444 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9jx\" (UniqueName: \"kubernetes.io/projected/22c76e55-4873-4a58-9ba5-c9cffedc1798-kube-api-access-4b9jx\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:43.997775 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:43.997744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:44.099296 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.099261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:44.300835 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.300754 2579 generic.go:356] "Generic (PLEG): container finished" podID="22c76e55-4873-4a58-9ba5-c9cffedc1798" containerID="3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f" exitCode=0 Sep 14 01:59:44.300835 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.300794 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" Sep 14 01:59:44.301013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.300844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" event={"ID":"22c76e55-4873-4a58-9ba5-c9cffedc1798","Type":"ContainerDied","Data":"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f"} Sep 14 01:59:44.301013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.300895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6" event={"ID":"22c76e55-4873-4a58-9ba5-c9cffedc1798","Type":"ContainerDied","Data":"ff45309524d3d254278917b5d5cea8cb03c300278eb4f373b659b192fd89400f"} Sep 14 01:59:44.301013 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.300923 2579 scope.go:117] "RemoveContainer" containerID="3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f" Sep 14 01:59:44.311298 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.311282 2579 scope.go:117] "RemoveContainer" containerID="3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f" Sep 14 01:59:44.311599 ip-10-0-130-123 kubenswrapper[2579]: E0914 01:59:44.311579 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f\": container with ID starting with 3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f not found: ID does not exist" containerID="3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f" Sep 14 01:59:44.311654 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.311611 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f"} err="failed to get container status \"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f\": rpc error: code = NotFound desc = could not find container \"3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f\": container with ID starting with 3ac6128f0e2cdc6c9a06e145d3c7f26f36dd84a1d8bf03782155e00737847b4f not found: ID does not exist" Sep 14 01:59:44.324842 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.324813 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:59:44.327992 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.327970 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ff93e-574dff7ffd-qdvl6"] Sep 14 01:59:44.693231 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.693138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:44.794227 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:44.794193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:45.210408 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:45.210372 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c76e55-4873-4a58-9ba5-c9cffedc1798" path="/var/lib/kubelet/pods/22c76e55-4873-4a58-9ba5-c9cffedc1798/volumes" Sep 14 01:59:45.210844 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:45.210822 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc40204-15d4-40f3-a565-3fdb48f82ffd" path="/var/lib/kubelet/pods/ffc40204-15d4-40f3-a565-3fdb48f82ffd/volumes" Sep 14 01:59:45.341394 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:45.341332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:46.117650 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:46.117617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:49.767580 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:49.767543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:49.868244 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:49.868208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:50.003893 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:50.003862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:55.462020 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.461979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:55.561129 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.561100 2579 scope.go:117] "RemoveContainer" containerID="281218cd9431aec4a4866ada2f92a20de51694490c35e55b879a4c7d8035f489" Sep 14 01:59:55.567252 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567228 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk"] Sep 14 01:59:55.567657 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567645 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c76e55-4873-4a58-9ba5-c9cffedc1798" containerName="payload-pre-processing" Sep 14 01:59:55.567701 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567660 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c76e55-4873-4a58-9ba5-c9cffedc1798" containerName="payload-pre-processing" Sep 14 01:59:55.567701 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567684 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc40204-15d4-40f3-a565-3fdb48f82ffd" containerName="revoke-keys" Sep 14 01:59:55.567701 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567690 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc40204-15d4-40f3-a565-3fdb48f82ffd" containerName="revoke-keys" Sep 14 01:59:55.567795 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567745 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc40204-15d4-40f3-a565-3fdb48f82ffd" containerName="revoke-keys" Sep 14 01:59:55.567795 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.567761 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c76e55-4873-4a58-9ba5-c9cffedc1798" containerName="payload-pre-processing" Sep 14 01:59:55.570465 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.570443 2579 scope.go:117] "RemoveContainer" containerID="96d3a84cc02754fb4570346a9d1ee937779ce4d78f0fdfd85ceeacbeaf7f9fb3" Sep 14 01:59:55.572337 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.572321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:55.574839 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.574820 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:59:55.575684 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.575666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 01:59:55.578278 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.578258 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk"] Sep 14 01:59:55.627343 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.627315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk\" (UID: \"9bed0d36-be52-4f4c-9ece-02eb8a381170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:55.728033 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.728006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk\" (UID: \"9bed0d36-be52-4f4c-9ece-02eb8a381170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:55.728568 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.728550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk\" (UID: \"9bed0d36-be52-4f4c-9ece-02eb8a381170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:55.885548 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:55.885515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:56.156486 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:56.156456 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk"] Sep 14 01:59:56.158294 ip-10-0-130-123 kubenswrapper[2579]: W0914 01:59:56.158267 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bed0d36_be52_4f4c_9ece_02eb8a381170.slice/crio-4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3 WatchSource:0}: Error finding container 4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3: Status 404 returned error can't find the container with id 4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3 Sep 14 01:59:56.313222 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:56.313189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:56.372563 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:56.372531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" event={"ID":"9bed0d36-be52-4f4c-9ece-02eb8a381170","Type":"ContainerStarted","Data":"4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3"} Sep 14 01:59:56.936970 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:56.936929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:57.383647 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:57.383611 2579 generic.go:356] "Generic (PLEG): container finished" podID="9bed0d36-be52-4f4c-9ece-02eb8a381170" containerID="96c148ff2bbdb3fc49abf6b237c338e8df00279c9390dd8e9a4444ffd1dedaa8" exitCode=0 Sep 14 01:59:57.383825 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:57.383701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" event={"ID":"9bed0d36-be52-4f4c-9ece-02eb8a381170","Type":"ContainerDied","Data":"96c148ff2bbdb3fc49abf6b237c338e8df00279c9390dd8e9a4444ffd1dedaa8"} Sep 14 01:59:57.550094 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:57.550038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:58.335290 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:58.335251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 01:59:58.534244 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:58.534221 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:58.655574 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:58.655494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca\") pod \"9bed0d36-be52-4f4c-9ece-02eb8a381170\" (UID: \"9bed0d36-be52-4f4c-9ece-02eb8a381170\") " Sep 14 01:59:58.655906 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:58.655878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bed0d36-be52-4f4c-9ece-02eb8a381170" (UID: "9bed0d36-be52-4f4c-9ece-02eb8a381170"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:58.757114 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:58.757083 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bed0d36-be52-4f4c-9ece-02eb8a381170-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 01:59:59.395920 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:59.395894 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" Sep 14 01:59:59.396336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:59.395932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk" event={"ID":"9bed0d36-be52-4f4c-9ece-02eb8a381170","Type":"ContainerDied","Data":"4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3"} Sep 14 01:59:59.396336 ip-10-0-130-123 kubenswrapper[2579]: I0914 01:59:59.395980 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5705a3ac0c125952b679b9fb90e1073f71b3472c769006a97e3084a7db77e3" Sep 14 02:00:00.026149 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.026107 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:00.129007 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.128970 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd"] Sep 14 02:00:00.129685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.129654 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bed0d36-be52-4f4c-9ece-02eb8a381170" containerName="revoke-keys" Sep 14 02:00:00.129685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.129680 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bed0d36-be52-4f4c-9ece-02eb8a381170" containerName="revoke-keys" Sep 14 02:00:00.129885 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.129824 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bed0d36-be52-4f4c-9ece-02eb8a381170" containerName="revoke-keys" Sep 14 02:00:00.131134 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.131090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:00.133497 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.133469 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898"] Sep 14 02:00:00.133595 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.133543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" Sep 14 02:00:00.135710 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.135692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:00:00.137684 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.137664 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2"] Sep 14 02:00:00.137804 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.137781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" Sep 14 02:00:00.141671 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.141649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd"] Sep 14 02:00:00.141800 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.141741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" Sep 14 02:00:00.143324 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.143217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898"] Sep 14 02:00:00.151406 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.151243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" Sep 14 02:00:00.158800 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.158771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2"] Sep 14 02:00:00.163033 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.163008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" Sep 14 02:00:00.202908 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.202882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" Sep 14 02:00:00.445853 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.445788 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 02:00:00.449534 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.449505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd"] Sep 14 02:00:00.452002 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:00:00.451971 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde065a5e_f36b_474b_b9ad_138690e35466.slice/crio-d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249 WatchSource:0}: Error finding container d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249: Status 404 returned error can't find the container with id d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249 Sep 14 02:00:00.452435 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.452410 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898"] Sep 14 02:00:00.453035 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:00:00.453011 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228fd95f_16f2_406d_978e_4db468d99355.slice/crio-83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0 WatchSource:0}: Error finding container 83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0: Status 404 returned error can't find the container with id 83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0 Sep 14 02:00:00.503811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.503782 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2"] Sep 14 02:00:00.505977 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:00:00.505950 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9327aa04_4760_435a_86f4_e1842332bc9b.slice/crio-95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d WatchSource:0}: Error finding container 95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d: Status 404 returned error can't find the container with id 95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d Sep 14 02:00:00.576663 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:00.576586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:01.221418 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.221323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:01.410994 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.410954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" event={"ID":"9327aa04-4760-435a-86f4-e1842332bc9b","Type":"ContainerStarted","Data":"93aa475bbc9cd802b82ee39f54cea83918b88e1be6b5c95209eaa4c4e4e38ec3"} Sep 14 02:00:01.411219 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.411023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" event={"ID":"9327aa04-4760-435a-86f4-e1842332bc9b","Type":"ContainerStarted","Data":"95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d"} Sep 14 02:00:01.413004 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.412801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" event={"ID":"de065a5e-f36b-474b-b9ad-138690e35466","Type":"ContainerStarted","Data":"62f922ab6a39aa5e991722d3f15f372f93e8797235fd8891a79f56bc4e511c3f"} Sep 14 02:00:01.413004 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.412848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" event={"ID":"de065a5e-f36b-474b-b9ad-138690e35466","Type":"ContainerStarted","Data":"d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249"} Sep 14 02:00:01.414560 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.414532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" event={"ID":"228fd95f-16f2-406d-978e-4db468d99355","Type":"ContainerStarted","Data":"3008349681db60f738c5dc2ab58ff2317b0caf40a26948d21840fc1f2e18e0f6"} Sep 14 02:00:01.414645 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.414585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" event={"ID":"228fd95f-16f2-406d-978e-4db468d99355","Type":"ContainerStarted","Data":"83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0"} Sep 14 02:00:01.426587 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.426543 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" podStartSLOduration=1.426530097 podStartE2EDuration="1.426530097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:00:01.424434716 +0000 UTC m=+1746.657799973" watchObservedRunningTime="2026-09-14 02:00:01.426530097 +0000 UTC m=+1746.659895380" Sep 14 02:00:01.441750 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.441699 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" podStartSLOduration=1.441685869 podStartE2EDuration="1.441685869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:00:01.43954346 +0000 UTC m=+1746.672908741" watchObservedRunningTime="2026-09-14 02:00:01.441685869 +0000 UTC m=+1746.675051125" Sep 14 02:00:01.454918 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:01.454873 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" podStartSLOduration=1.4548603500000001 podStartE2EDuration="1.45486035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:00:01.454045618 +0000 UTC m=+1746.687410875" watchObservedRunningTime="2026-09-14 02:00:01.45486035 +0000 UTC m=+1746.688225607" Sep 14 02:00:02.422624 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.422589 2579 generic.go:356] "Generic (PLEG): container finished" podID="9327aa04-4760-435a-86f4-e1842332bc9b" containerID="93aa475bbc9cd802b82ee39f54cea83918b88e1be6b5c95209eaa4c4e4e38ec3" exitCode=0 Sep 14 02:00:02.422791 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.422672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" event={"ID":"9327aa04-4760-435a-86f4-e1842332bc9b","Type":"ContainerDied","Data":"93aa475bbc9cd802b82ee39f54cea83918b88e1be6b5c95209eaa4c4e4e38ec3"} Sep 14 02:00:02.424158 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.424138 2579 generic.go:356] "Generic (PLEG): container finished" podID="de065a5e-f36b-474b-b9ad-138690e35466" containerID="62f922ab6a39aa5e991722d3f15f372f93e8797235fd8891a79f56bc4e511c3f" exitCode=0 Sep 14 02:00:02.424276 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.424208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" event={"ID":"de065a5e-f36b-474b-b9ad-138690e35466","Type":"ContainerDied","Data":"62f922ab6a39aa5e991722d3f15f372f93e8797235fd8891a79f56bc4e511c3f"} Sep 14 02:00:02.425624 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.425604 2579 generic.go:356] "Generic (PLEG): container finished" podID="228fd95f-16f2-406d-978e-4db468d99355" containerID="3008349681db60f738c5dc2ab58ff2317b0caf40a26948d21840fc1f2e18e0f6" exitCode=0 Sep 14 02:00:02.425723 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:02.425622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" event={"ID":"228fd95f-16f2-406d-978e-4db468d99355","Type":"ContainerDied","Data":"3008349681db60f738c5dc2ab58ff2317b0caf40a26948d21840fc1f2e18e0f6"} Sep 14 02:00:03.571850 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:03.571825 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" Sep 14 02:00:03.618718 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:03.618692 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" Sep 14 02:00:03.638860 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:03.638838 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" Sep 14 02:00:04.438558 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.438530 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" Sep 14 02:00:04.438558 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.438545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2" event={"ID":"9327aa04-4760-435a-86f4-e1842332bc9b","Type":"ContainerDied","Data":"95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d"} Sep 14 02:00:04.438807 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.438595 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95606da79710f51a86c7a7642ba75540e65b816def36e2146082b444f88cd23d" Sep 14 02:00:04.439861 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.439843 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" Sep 14 02:00:04.439970 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.439876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29822520-zj898" event={"ID":"de065a5e-f36b-474b-b9ad-138690e35466","Type":"ContainerDied","Data":"d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249"} Sep 14 02:00:04.439970 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.439915 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92a8bcff2d611d85df78ea2acb818b63c47a82220cb3bcd14931f7bea1b0249" Sep 14 02:00:04.441180 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.441163 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" Sep 14 02:00:04.441180 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.441168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd" event={"ID":"228fd95f-16f2-406d-978e-4db468d99355","Type":"ContainerDied","Data":"83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0"} Sep 14 02:00:04.441367 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:04.441200 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e9fe9fa38c829ecb1c8059adff696779291467a4618f38c3986394c6cd73e0" Sep 14 02:00:05.890474 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:05.890431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.296100 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.296045 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2"] Sep 14 02:00:11.299595 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.299568 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8e5935-29822520-m4dz2"] Sep 14 02:00:11.319175 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.319140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 02:00:11.319443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.319414 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" podUID="277ca8a3-1391-4c51-89cb-95590ea55b71" containerName="payload-processing" containerID="cri-o://3ab25ac81a45d0fd17e7ed0c1e85d6d9bb2f39d721966d3369d0be339cb57578" gracePeriod=30 Sep 14 02:00:11.324827 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.324730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.325528 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.325502 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 02:00:11.325746 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.325718 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" podUID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" containerName="payload-pre-processing" containerID="cri-o://4e8b50e703862d92588242479e2192e9f491630467d7d3de9878c144bb1243ee" gracePeriod=30 Sep 14 02:00:11.384379 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.384338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.476584 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.476536 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk"] Sep 14 02:00:11.478754 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.478725 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8e5935-7k5sk"] Sep 14 02:00:11.483322 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.483292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.485734 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.485706 2579 generic.go:356] "Generic (PLEG): container finished" podID="277ca8a3-1391-4c51-89cb-95590ea55b71" containerID="3ab25ac81a45d0fd17e7ed0c1e85d6d9bb2f39d721966d3369d0be339cb57578" exitCode=0 Sep 14 02:00:11.485853 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.485777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" event={"ID":"277ca8a3-1391-4c51-89cb-95590ea55b71","Type":"ContainerDied","Data":"3ab25ac81a45d0fd17e7ed0c1e85d6d9bb2f39d721966d3369d0be339cb57578"} Sep 14 02:00:11.488005 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.487983 2579 generic.go:356] "Generic (PLEG): container finished" podID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" containerID="4e8b50e703862d92588242479e2192e9f491630467d7d3de9878c144bb1243ee" exitCode=0 Sep 14 02:00:11.488110 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.488036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" event={"ID":"38688eb8-d5c1-4594-aca2-6c676f3b61dc","Type":"ContainerDied","Data":"4e8b50e703862d92588242479e2192e9f491630467d7d3de9878c144bb1243ee"} Sep 14 02:00:11.536551 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.536516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.630216 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.630184 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 02:00:11.632960 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.632942 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 02:00:11.781583 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.781549 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume\") pod \"277ca8a3-1391-4c51-89cb-95590ea55b71\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " Sep 14 02:00:11.781743 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.781599 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jknvq\" (UniqueName: \"kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq\") pod \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " Sep 14 02:00:11.781743 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.781632 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44pps\" (UniqueName: \"kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps\") pod \"277ca8a3-1391-4c51-89cb-95590ea55b71\" (UID: \"277ca8a3-1391-4c51-89cb-95590ea55b71\") " Sep 14 02:00:11.781743 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.781673 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume\") pod \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\" (UID: \"38688eb8-d5c1-4594-aca2-6c676f3b61dc\") " Sep 14 02:00:11.782071 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.782024 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "277ca8a3-1391-4c51-89cb-95590ea55b71" (UID: "277ca8a3-1391-4c51-89cb-95590ea55b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:00:11.782171 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.782132 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38688eb8-d5c1-4594-aca2-6c676f3b61dc" (UID: "38688eb8-d5c1-4594-aca2-6c676f3b61dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:00:11.783867 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.783846 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps" (OuterVolumeSpecName: "kube-api-access-44pps") pod "277ca8a3-1391-4c51-89cb-95590ea55b71" (UID: "277ca8a3-1391-4c51-89cb-95590ea55b71"). InnerVolumeSpecName "kube-api-access-44pps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:11.783914 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.783879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq" (OuterVolumeSpecName: "kube-api-access-jknvq") pod "38688eb8-d5c1-4594-aca2-6c676f3b61dc" (UID: "38688eb8-d5c1-4594-aca2-6c676f3b61dc"). InnerVolumeSpecName "kube-api-access-jknvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:11.808757 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.808724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:11.882899 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.882826 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277ca8a3-1391-4c51-89cb-95590ea55b71-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.882899 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.882854 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jknvq\" (UniqueName: \"kubernetes.io/projected/38688eb8-d5c1-4594-aca2-6c676f3b61dc-kube-api-access-jknvq\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.882899 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.882863 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pps\" (UniqueName: \"kubernetes.io/projected/277ca8a3-1391-4c51-89cb-95590ea55b71-kube-api-access-44pps\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.882899 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.882871 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38688eb8-d5c1-4594-aca2-6c676f3b61dc-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.932212 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:11.932182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:12.495642 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.495608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" Sep 14 02:00:12.496097 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.495645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg" event={"ID":"277ca8a3-1391-4c51-89cb-95590ea55b71","Type":"ContainerDied","Data":"d33b9b9ec8597cf97740988d1f54101f40e69b641fb0a5126cf6f8c87424a0e5"} Sep 14 02:00:12.496097 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.495711 2579 scope.go:117] "RemoveContainer" containerID="3ab25ac81a45d0fd17e7ed0c1e85d6d9bb2f39d721966d3369d0be339cb57578" Sep 14 02:00:12.497167 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.497151 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" Sep 14 02:00:12.497261 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.497183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b" event={"ID":"38688eb8-d5c1-4594-aca2-6c676f3b61dc","Type":"ContainerDied","Data":"74f13cc75704e0fc56c870c22f367e91f96a5090d96fe238ba7941633855b01d"} Sep 14 02:00:12.506207 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.506184 2579 scope.go:117] "RemoveContainer" containerID="4e8b50e703862d92588242479e2192e9f491630467d7d3de9878c144bb1243ee" Sep 14 02:00:12.520772 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.520748 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 02:00:12.524914 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.524887 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8e5935-77769c84f8-t484b"] Sep 14 02:00:12.538997 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.538970 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 02:00:12.542163 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.542140 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8e5935-d88d756f-4sccg"] Sep 14 02:00:12.735669 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:12.735634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:13.211100 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:13.211049 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="277ca8a3-1391-4c51-89cb-95590ea55b71" path="/var/lib/kubelet/pods/277ca8a3-1391-4c51-89cb-95590ea55b71/volumes" Sep 14 02:00:13.211454 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:13.211438 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" path="/var/lib/kubelet/pods/38688eb8-d5c1-4594-aca2-6c676f3b61dc/volumes" Sep 14 02:00:13.211688 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:13.211675 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9327aa04-4760-435a-86f4-e1842332bc9b" path="/var/lib/kubelet/pods/9327aa04-4760-435a-86f4-e1842332bc9b/volumes" Sep 14 02:00:13.211934 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:13.211920 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bed0d36-be52-4f4c-9ece-02eb8a381170" path="/var/lib/kubelet/pods/9bed0d36-be52-4f4c-9ece-02eb8a381170/volumes" Sep 14 02:00:13.353476 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:13.353428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:15.467383 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.467331 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" podUID="823e58cd-d4e0-45f7-9751-e17b87cbb030" containerName="main" containerID="cri-o://0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd" gracePeriod=45 Sep 14 02:00:15.722938 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.722884 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 02:00:15.919895 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.919855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.919903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.919951 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.919972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920013 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjnx7\" (UniqueName: \"kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7\") pod \"823e58cd-d4e0-45f7-9751-e17b87cbb030\" (UID: \"823e58cd-d4e0-45f7-9751-e17b87cbb030\") " Sep 14 02:00:15.920373 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920168 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:15.920373 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home" (OuterVolumeSpecName: "home") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:15.920373 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache" (OuterVolumeSpecName: "model-cache") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:15.920525 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920433 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-tmp-dir\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:15.920525 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920454 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-model-cache\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:15.920525 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.920467 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-home\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:15.922100 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.922054 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:00:15.922231 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.922109 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm" (OuterVolumeSpecName: "dshm") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:15.922291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:15.922221 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7" (OuterVolumeSpecName: "kube-api-access-bjnx7") pod "823e58cd-d4e0-45f7-9751-e17b87cbb030" (UID: "823e58cd-d4e0-45f7-9751-e17b87cbb030"). InnerVolumeSpecName "kube-api-access-bjnx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:16.021844 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.021808 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823e58cd-d4e0-45f7-9751-e17b87cbb030-dshm\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:16.021844 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.021838 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823e58cd-d4e0-45f7-9751-e17b87cbb030-tls-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:16.022088 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.021852 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnx7\" (UniqueName: \"kubernetes.io/projected/823e58cd-d4e0-45f7-9751-e17b87cbb030-kube-api-access-bjnx7\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:16.524286 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.524253 2579 generic.go:356] "Generic (PLEG): container finished" podID="823e58cd-d4e0-45f7-9751-e17b87cbb030" containerID="0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd" exitCode=0 Sep 14 02:00:16.524647 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.524298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" Sep 14 02:00:16.524647 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.524315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" event={"ID":"823e58cd-d4e0-45f7-9751-e17b87cbb030","Type":"ContainerDied","Data":"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd"} Sep 14 02:00:16.524647 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.524356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8" event={"ID":"823e58cd-d4e0-45f7-9751-e17b87cbb030","Type":"ContainerDied","Data":"91840d1b5c92e32f6a2ffe77b9d09462ef7397df140e4992121e924bcad719c3"} Sep 14 02:00:16.524647 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.524372 2579 scope.go:117] "RemoveContainer" containerID="0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd" Sep 14 02:00:16.554254 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.533853 2579 scope.go:117] "RemoveContainer" containerID="0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd" Sep 14 02:00:16.554254 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:00:16.534163 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd\": container with ID starting with 0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd not found: ID does not exist" containerID="0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd" Sep 14 02:00:16.554254 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.534194 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd"} err="failed to get container status \"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd\": rpc error: code = NotFound desc = could not find container \"0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd\": container with ID starting with 0931e4e8032c6cd93b7747e5780992efa0437e7bd272b38f5c5882dcbcd744fd not found: ID does not exist" Sep 14 02:00:16.554254 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.546700 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 02:00:16.554254 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:16.551428 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8e5935/distinct-2-w4-8e5935-kserve-66dd98b5b9-4c4s8"] Sep 14 02:00:17.211130 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.211097 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823e58cd-d4e0-45f7-9751-e17b87cbb030" path="/var/lib/kubelet/pods/823e58cd-d4e0-45f7-9751-e17b87cbb030/volumes" Sep 14 02:00:17.579869 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.579827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:17.640770 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.640739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:17.807673 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.807636 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 02:00:17.807947 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.807901 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" containerID="cri-o://349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663" gracePeriod=30 Sep 14 02:00:17.826214 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:17.826180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:20.427668 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:20.427626 2579 patch_prober.go:28] interesting pod/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 02:00:20.428047 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:20.427701 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 02:00:23.040372 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.040332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:23.062729 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.062706 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 02:00:23.085800 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.085943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.085943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.085943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085874 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.085943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.086192 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.086192 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.085983 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtvtv\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.086192 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086045 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.086192 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086092 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo\") pod \"f2088d45-5b19-4cf9-a333-978ceb60d588\" (UID: \"f2088d45-5b19-4cf9-a333-978ceb60d588\") " Sep 14 02:00:23.086439 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086411 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:23.086813 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086772 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:23.086959 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:00:23.087044 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.086854 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:23.087219 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.087197 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:23.088723 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.088695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:00:23.088959 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.088856 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:23.088959 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.088932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv" (OuterVolumeSpecName: "kube-api-access-mtvtv") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "kube-api-access-mtvtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:23.089507 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.089485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2088d45-5b19-4cf9-a333-978ceb60d588" (UID: "f2088d45-5b19-4cf9-a333-978ceb60d588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:00:23.187450 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187415 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187450 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187444 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187450 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187454 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187462 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2088d45-5b19-4cf9-a333-978ceb60d588-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187471 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187479 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvtv\" (UniqueName: \"kubernetes.io/projected/f2088d45-5b19-4cf9-a333-978ceb60d588-kube-api-access-mtvtv\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187488 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187496 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2088d45-5b19-4cf9-a333-978ceb60d588-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.187685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.187512 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2088d45-5b19-4cf9-a333-978ceb60d588-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.566165 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.566132 2579 generic.go:356] "Generic (PLEG): container finished" podID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerID="349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663" exitCode=0 Sep 14 02:00:23.566334 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.566177 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" Sep 14 02:00:23.566334 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.566219 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerDied","Data":"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663"} Sep 14 02:00:23.566334 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.566277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85" event={"ID":"f2088d45-5b19-4cf9-a333-978ceb60d588","Type":"ContainerDied","Data":"e1370bd6aa18e404fb89a579cf9a587400bbe975817cdc1d8d8056176814b5d9"} Sep 14 02:00:23.566334 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.566300 2579 scope.go:117] "RemoveContainer" containerID="349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663" Sep 14 02:00:23.575444 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.575427 2579 scope.go:117] "RemoveContainer" containerID="cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416" Sep 14 02:00:23.583919 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.583890 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 02:00:23.584887 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.584866 2579 scope.go:117] "RemoveContainer" containerID="349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663" Sep 14 02:00:23.585428 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:00:23.585405 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663\": container with ID starting with 349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663 not found: ID does not exist" containerID="349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663" Sep 14 02:00:23.585526 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.585435 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663"} err="failed to get container status \"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663\": rpc error: code = NotFound desc = could not find container \"349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663\": container with ID starting with 349e175b3e0132098f34530d506c13a614640c6490f5f53433c5885a85f58663 not found: ID does not exist" Sep 14 02:00:23.585526 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.585454 2579 scope.go:117] "RemoveContainer" containerID="cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416" Sep 14 02:00:23.585737 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:00:23.585718 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416\": container with ID starting with cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416 not found: ID does not exist" containerID="cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416" Sep 14 02:00:23.585784 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.585742 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416"} err="failed to get container status \"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416\": rpc error: code = NotFound desc = could not find container \"cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416\": container with ID starting with cb1e5f2c3b2a7e123599debf1b612469418eba078c506f43ccc3f42334e17416 not found: ID does not exist" Sep 14 02:00:23.586961 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:23.586941 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8e5935-openshift-default-5dc7bb4f58-bvx85"] Sep 14 02:00:24.080589 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:24.080552 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:24.178345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:24.178312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:24.933781 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:24.933742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:25.214709 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:25.214616 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" path="/var/lib/kubelet/pods/f2088d45-5b19-4cf9-a333-978ceb60d588/volumes" Sep 14 02:00:29.346359 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:29.346327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:35.952035 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:35.952001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:36.007775 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:36.007738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:36.084494 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:36.084438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:50.166858 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:50.166765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:00:55.318651 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:55.318612 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:00:55.325107 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:55.325079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:00:58.216132 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:00:58.216083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:01:45.457438 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:01:45.457406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:01:46.888824 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:01:46.888781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:01:51.033364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:01:51.033324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:01:55.665724 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:01:55.665690 2579 scope.go:117] "RemoveContainer" containerID="112dfa85d13fc93bc90f525b35b22413d549986316ec9bda6ef9218a3422e887" Sep 14 02:02:17.395557 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395522 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395913 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de065a5e-f36b-474b-b9ad-138690e35466" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395924 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de065a5e-f36b-474b-b9ad-138690e35466" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395936 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395941 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395948 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" containerName="payload-pre-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395953 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" containerName="payload-pre-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395964 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="277ca8a3-1391-4c51-89cb-95590ea55b71" containerName="payload-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395969 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="277ca8a3-1391-4c51-89cb-95590ea55b71" containerName="payload-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395983 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9327aa04-4760-435a-86f4-e1842332bc9b" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395988 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9327aa04-4760-435a-86f4-e1842332bc9b" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.395996 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823e58cd-d4e0-45f7-9751-e17b87cbb030" containerName="main" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396001 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="823e58cd-d4e0-45f7-9751-e17b87cbb030" containerName="main" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396009 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228fd95f-16f2-406d-978e-4db468d99355" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396013 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="228fd95f-16f2-406d-978e-4db468d99355" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396086 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de065a5e-f36b-474b-b9ad-138690e35466" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396094 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="228fd95f-16f2-406d-978e-4db468d99355" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396101 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396107 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396113 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="277ca8a3-1391-4c51-89cb-95590ea55b71" containerName="payload-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396119 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9327aa04-4760-435a-86f4-e1842332bc9b" containerName="cleanup" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396126 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38688eb8-d5c1-4594-aca2-6c676f3b61dc" containerName="payload-pre-processing" Sep 14 02:02:17.396188 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.396131 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="823e58cd-d4e0-45f7-9751-e17b87cbb030" containerName="main" Sep 14 02:02:17.399440 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.399419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.401835 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.401812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-86995c\"/\"kube-root-ca.crt\"" Sep 14 02:02:17.403053 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.403032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-86995c\"/\"rate-test-model-86995c-kserve-self-signed-certs\"" Sep 14 02:02:17.403187 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.403136 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-86995c\"/\"openshift-service-ca.crt\"" Sep 14 02:02:17.403187 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.403146 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-86995c\"/\"default-dockercfg-5gscb\"" Sep 14 02:02:17.408861 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.408824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:02:17.430397 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.430370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:02:17.556575 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwxn\" (UniqueName: \"kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.556775 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.556775 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.556775 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.556936 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.556936 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.556827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.657790 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.657790 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658022 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwxn\" (UniqueName: \"kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658022 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658022 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658022 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.657918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658276 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.658209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658338 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.658298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.658338 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.658319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.660273 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.660249 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.660436 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.660419 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.665840 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.665816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwxn\" (UniqueName: \"kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn\") pod \"rate-test-model-86995c-kserve-7bd644f6f5-gblns\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.712131 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.712108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:17.982910 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:17.982881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:02:17.985352 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:02:17.985325 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef2ef763_a00f_4e89_b542_62f494cfd9a9.slice/crio-2bed5982e9b755ccfc87c3261b0275a397dcababfdd9e327e1cdc866a9788993 WatchSource:0}: Error finding container 2bed5982e9b755ccfc87c3261b0275a397dcababfdd9e327e1cdc866a9788993: Status 404 returned error can't find the container with id 2bed5982e9b755ccfc87c3261b0275a397dcababfdd9e327e1cdc866a9788993 Sep 14 02:02:18.214858 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:18.214822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" event={"ID":"ef2ef763-a00f-4e89-b542-62f494cfd9a9","Type":"ContainerStarted","Data":"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8"} Sep 14 02:02:18.215052 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:18.214869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" event={"ID":"ef2ef763-a00f-4e89-b542-62f494cfd9a9","Type":"ContainerStarted","Data":"2bed5982e9b755ccfc87c3261b0275a397dcababfdd9e327e1cdc866a9788993"} Sep 14 02:02:18.231737 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:18.231686 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" podStartSLOduration=1.231670841 podStartE2EDuration="1.231670841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:02:18.229594309 +0000 UTC m=+1883.462959566" watchObservedRunningTime="2026-09-14 02:02:18.231670841 +0000 UTC m=+1883.465036097" Sep 14 02:02:27.713078 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:27.713017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:27.713554 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:27.713104 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:27.725978 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:27.725952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:02:28.282011 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:02:28.281985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:03:00.141219 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:00.141186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:01.106164 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:01.106127 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:03:01.119571 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:01.119509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:01.558336 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:01.558298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:01.853624 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:01.853516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:02.253610 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:02.253560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:04.558306 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:04.558271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:05.321070 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:05.321016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:05.420995 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:05.420961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:06.238726 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:06.238687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.008748 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.008708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.066943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.066906 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.165736 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.165692 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.548314 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.548276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.652752 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.652716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:07.654359 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.654338 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:03:07.654762 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.654751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:03:07.654807 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.654764 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2088d45-5b19-4cf9-a333-978ceb60d588" containerName="istio-proxy" Sep 14 02:03:07.657838 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.657823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.660341 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.660325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-39c298\"" Sep 14 02:03:07.660447 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.660346 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-39c298-dockercfg-792sp\"" Sep 14 02:03:07.668602 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.668583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:03:07.828566 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.828478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.828717 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.828585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frj4\" (UniqueName: \"kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.929719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.929682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frj4\" (UniqueName: \"kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.929883 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.929798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.930503 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.930484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.937422 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.937401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frj4\" (UniqueName: \"kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4\") pod \"payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:07.969351 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:07.969308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:08.241618 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:08.241597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:03:08.243732 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:03:08.243705 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae94c80_6a55_43ea_94b1_60c7aefa8d8d.slice/crio-1f8ac627a5a4addb906fa925cf7fce935436166fa69d2c6e16bf461a4e3ff411 WatchSource:0}: Error finding container 1f8ac627a5a4addb906fa925cf7fce935436166fa69d2c6e16bf461a4e3ff411: Status 404 returned error can't find the container with id 1f8ac627a5a4addb906fa925cf7fce935436166fa69d2c6e16bf461a4e3ff411 Sep 14 02:03:08.508723 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:08.508616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" event={"ID":"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d","Type":"ContainerStarted","Data":"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d"} Sep 14 02:03:08.508723 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:08.508676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:08.508723 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:08.508696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" event={"ID":"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d","Type":"ContainerStarted","Data":"1f8ac627a5a4addb906fa925cf7fce935436166fa69d2c6e16bf461a4e3ff411"} Sep 14 02:03:08.525438 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:08.525386 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" podStartSLOduration=1.525371045 podStartE2EDuration="1.525371045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:08.522430503 +0000 UTC m=+1933.755795784" watchObservedRunningTime="2026-09-14 02:03:08.525371045 +0000 UTC m=+1933.758736301" Sep 14 02:03:13.037732 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:13.037696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:16.125891 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.125851 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" podUID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" containerName="main" containerID="cri-o://538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8" gracePeriod=45 Sep 14 02:03:16.376601 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.376543 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:03:16.415207 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415166 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415207 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415203 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415463 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415463 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415285 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415576 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415491 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:03:16.415576 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415502 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home" (OuterVolumeSpecName: "home") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:03:16.415576 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415378 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cwxn\" (UniqueName: \"kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415806 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415571 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:03:16.415806 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm\") pod \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\" (UID: \"ef2ef763-a00f-4e89-b542-62f494cfd9a9\") " Sep 14 02:03:16.415972 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.415949 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-home\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.416172 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.416151 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-model-cache\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.416252 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.416175 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tmp-dir\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.417823 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.417800 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm" (OuterVolumeSpecName: "dshm") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:03:16.417980 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.417963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn" (OuterVolumeSpecName: "kube-api-access-5cwxn") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "kube-api-access-5cwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:03:16.418131 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.418025 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef2ef763-a00f-4e89-b542-62f494cfd9a9" (UID: "ef2ef763-a00f-4e89-b542-62f494cfd9a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:03:16.517277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.517240 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2ef763-a00f-4e89-b542-62f494cfd9a9-tls-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.517277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.517273 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwxn\" (UniqueName: \"kubernetes.io/projected/ef2ef763-a00f-4e89-b542-62f494cfd9a9-kube-api-access-5cwxn\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.517277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.517284 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2ef763-a00f-4e89-b542-62f494cfd9a9-dshm\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:03:16.553582 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.553549 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" containerID="538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8" exitCode=0 Sep 14 02:03:16.553719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.553595 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" Sep 14 02:03:16.553719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.553635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" event={"ID":"ef2ef763-a00f-4e89-b542-62f494cfd9a9","Type":"ContainerDied","Data":"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8"} Sep 14 02:03:16.553719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.553674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns" event={"ID":"ef2ef763-a00f-4e89-b542-62f494cfd9a9","Type":"ContainerDied","Data":"2bed5982e9b755ccfc87c3261b0275a397dcababfdd9e327e1cdc866a9788993"} Sep 14 02:03:16.553719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.553692 2579 scope.go:117] "RemoveContainer" containerID="538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8" Sep 14 02:03:16.563565 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.563548 2579 scope.go:117] "RemoveContainer" containerID="538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8" Sep 14 02:03:16.563805 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:03:16.563788 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8\": container with ID starting with 538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8 not found: ID does not exist" containerID="538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8" Sep 14 02:03:16.563857 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.563814 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8"} err="failed to get container status \"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8\": rpc error: code = NotFound desc = could not find container \"538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8\": container with ID starting with 538d75f9e2636bdb0b8c7c5526c7c043026d6e2a8fc21ef99458677cf2efa1c8 not found: ID does not exist" Sep 14 02:03:16.578044 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.578018 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:03:16.580069 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:16.580040 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-86995c/rate-test-model-86995c-kserve-7bd644f6f5-gblns"] Sep 14 02:03:17.210943 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:17.210910 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" path="/var/lib/kubelet/pods/ef2ef763-a00f-4e89-b542-62f494cfd9a9/volumes" Sep 14 02:03:19.517016 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:19.516985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:03:26.545090 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:26.545037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:28.806481 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:28.806448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:29.931017 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:29.930968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:30.356899 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:30.356868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:30.722700 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:30.722607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:31.097447 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.097413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:31.382746 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.382657 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:03:31.383240 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.383217 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" containerName="main" Sep 14 02:03:31.383240 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.383241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" containerName="main" Sep 14 02:03:31.383402 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.383380 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef2ef763-a00f-4e89-b542-62f494cfd9a9" containerName="main" Sep 14 02:03:31.386683 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.386662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.389277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.389257 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8020f1-openshift-default-dockercfg-dzjz9\"" Sep 14 02:03:31.398175 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.398150 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:03:31.416243 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.416207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:31.462352 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462352 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462386 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2sd\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462572 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.462778 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.462618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.563722 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.563684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.563722 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.563727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.563977 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.563750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.563977 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.563881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.563977 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.563968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2sd\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564492 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564492 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564606 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.564855 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.564834 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.565983 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.565966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.566377 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.566359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.577969 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.577945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.578300 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.578284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2sd\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd\") pod \"e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.602028 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.601984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:31.699911 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.699827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:31.737282 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.737249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:31.981408 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.981384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:03:31.982750 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:03:31.982725 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ec6501_01d3_4ea2_894e_399f6873534c.slice/crio-dbc714c4e8e97a27ec05502c65dae236819d13c5e6b7dc7e9a974cbea3360508 WatchSource:0}: Error finding container dbc714c4e8e97a27ec05502c65dae236819d13c5e6b7dc7e9a974cbea3360508: Status 404 returned error can't find the container with id dbc714c4e8e97a27ec05502c65dae236819d13c5e6b7dc7e9a974cbea3360508 Sep 14 02:03:31.985091 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.985049 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:31.985158 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.985125 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:31.985195 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:31.985162 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:32.254838 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:32.254754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:32.656384 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:32.656286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerStarted","Data":"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea"} Sep 14 02:03:32.656384 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:32.656349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerStarted","Data":"dbc714c4e8e97a27ec05502c65dae236819d13c5e6b7dc7e9a974cbea3360508"} Sep 14 02:03:32.675149 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:32.675078 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podStartSLOduration=1.675045654 podStartE2EDuration="1.675045654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:32.674244361 +0000 UTC m=+1957.907609638" watchObservedRunningTime="2026-09-14 02:03:32.675045654 +0000 UTC m=+1957.908410912" Sep 14 02:03:32.700620 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:32.700572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:33.705886 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:33.705849 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:33.917699 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:33.917664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:34.531114 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:34.531083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:34.632695 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:34.632655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:34.667810 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:34.667777 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:34.668920 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:34.668899 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:34.731730 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:34.731698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:35.439033 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:35.438993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:36.064460 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:36.064426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:36.838107 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:36.838047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:36.890742 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:36.890704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:37.048420 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.048375 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:37.295125 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.295089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:37.376832 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.376789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:37.381315 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.381286 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:03:37.384916 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.384899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.387757 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.387737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8020f1-dockercfg-8n7ng\"" Sep 14 02:03:37.387862 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.387797 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8020f1\"" Sep 14 02:03:37.399396 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.399371 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:03:37.412098 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.412046 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:03:37.416591 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.416561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.422294 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.422269 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:03:37.523737 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.523700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgxj\" (UniqueName: \"kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.523916 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.523760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.523916 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.523802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spcj\" (UniqueName: \"kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.523996 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.523922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.624655 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.624567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.624655 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.624623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgxj\" (UniqueName: \"kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.624655 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.624656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.624884 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.624679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spcj\" (UniqueName: \"kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.625363 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.625344 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.625465 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.625448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.632862 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.632832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spcj\" (UniqueName: \"kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj\") pod \"payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.632976 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.632955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgxj\" (UniqueName: \"kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj\") pod \"payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.695884 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.695848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:37.729431 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.729395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:37.985622 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:37.985595 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:03:37.987264 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:03:37.987241 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64225315_143b_42c1_8d55_642270d7b2d7.slice/crio-75d0161d489cee07ac55eb7438b493efc714f64a2a0dfe87eda2b5235708a543 WatchSource:0}: Error finding container 75d0161d489cee07ac55eb7438b493efc714f64a2a0dfe87eda2b5235708a543: Status 404 returned error can't find the container with id 75d0161d489cee07ac55eb7438b493efc714f64a2a0dfe87eda2b5235708a543 Sep 14 02:03:38.021591 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.021560 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:03:38.023083 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:03:38.023039 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26634479_ceea_44e8_9471_69fffb0c337d.slice/crio-988d460211a1c9a49f6fb541ed62d070ac145c470920f8dd61affd1dc2a7f096 WatchSource:0}: Error finding container 988d460211a1c9a49f6fb541ed62d070ac145c470920f8dd61affd1dc2a7f096: Status 404 returned error can't find the container with id 988d460211a1c9a49f6fb541ed62d070ac145c470920f8dd61affd1dc2a7f096 Sep 14 02:03:38.692228 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.692187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" event={"ID":"64225315-143b-42c1-8d55-642270d7b2d7","Type":"ContainerStarted","Data":"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6"} Sep 14 02:03:38.692667 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.692260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" event={"ID":"64225315-143b-42c1-8d55-642270d7b2d7","Type":"ContainerStarted","Data":"75d0161d489cee07ac55eb7438b493efc714f64a2a0dfe87eda2b5235708a543"} Sep 14 02:03:38.692667 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.692294 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:38.693957 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.693929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" event={"ID":"26634479-ceea-44e8-9471-69fffb0c337d","Type":"ContainerStarted","Data":"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3"} Sep 14 02:03:38.694105 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.693975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" event={"ID":"26634479-ceea-44e8-9471-69fffb0c337d","Type":"ContainerStarted","Data":"988d460211a1c9a49f6fb541ed62d070ac145c470920f8dd61affd1dc2a7f096"} Sep 14 02:03:38.694105 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.694018 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:38.708454 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.708409 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" podStartSLOduration=1.7083857409999998 podStartE2EDuration="1.708385741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:38.707265403 +0000 UTC m=+1963.940630663" watchObservedRunningTime="2026-09-14 02:03:38.708385741 +0000 UTC m=+1963.941750997" Sep 14 02:03:38.725541 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:38.725487 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" podStartSLOduration=1.725470002 podStartE2EDuration="1.725470002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:38.721839757 +0000 UTC m=+1963.955205010" watchObservedRunningTime="2026-09-14 02:03:38.725470002 +0000 UTC m=+1963.958835261" Sep 14 02:03:49.701386 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:49.701350 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:03:49.701761 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:49.701590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:03:50.710248 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:50.710210 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 02:03:50.710620 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:50.710270 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 02:03:51.779120 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.779087 2579 generic.go:356] "Generic (PLEG): container finished" podID="04ec6501-01d3-4ea2-894e-399f6873534c" containerID="38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea" exitCode=0 Sep 14 02:03:51.779520 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.779165 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerDied","Data":"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea"} Sep 14 02:03:51.779567 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.779550 2579 scope.go:117] "RemoveContainer" containerID="38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea" Sep 14 02:03:51.780656 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.780631 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:51.780714 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.780702 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:51.780751 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:51.780735 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:03:52.787667 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:52.787630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerStarted","Data":"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8"} Sep 14 02:03:53.700716 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:53.700679 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:53.700929 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:53.700901 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Sep 14 02:03:53.700996 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:53.700969 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Sep 14 02:03:54.701248 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:54.701213 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Sep 14 02:03:54.701641 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:54.701276 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Sep 14 02:03:55.700425 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:55.700388 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Sep 14 02:03:55.700611 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:55.700452 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Sep 14 02:03:56.700528 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:56.700484 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Sep 14 02:03:56.700909 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:56.700544 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Sep 14 02:03:57.700695 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:57.700657 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Sep 14 02:03:57.701107 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:57.700719 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Sep 14 02:03:58.701672 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:58.701623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:58.704529 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:58.704505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:58.704900 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:58.704877 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:58.706130 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:58.706106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:03:59.306894 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:59.306852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:59.632010 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:59.631919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:03:59.935124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:03:59.935029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:00.733200 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:00.733154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:01.450986 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:01.450953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:04.855291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:04.855253 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj_26634479-ceea-44e8-9471-69fffb0c337d/payload-processing/0.log" Sep 14 02:04:20.334079 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:20.333972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:30.275592 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:30.275546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:43.607919 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:43.607865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:04:55.710626 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:55.710595 2579 scope.go:117] "RemoveContainer" containerID="1df63fa9d6018d3f828f6baac7aaba39f345c3fd8d876674ab3fce3a06d368a1" Sep 14 02:04:55.719241 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:55.719222 2579 scope.go:117] "RemoveContainer" containerID="f0a9d6d08a6e2ecc5531b8a10e7949c9e293fb730da025b1065e7ab93e955203" Sep 14 02:04:55.726820 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:04:55.726800 2579 scope.go:117] "RemoveContainer" containerID="1f84751df2a62212a2750fb797466908b15fd4f734df8b8711c4b49e8fcf5578" Sep 14 02:05:01.950835 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:01.950796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:06.523418 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:06.523386 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj_26634479-ceea-44e8-9471-69fffb0c337d/payload-processing/0.log" Sep 14 02:05:06.806200 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:06.806104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:07.079305 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.079216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:05:07.083309 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.083287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.085721 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.085700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-072cc4-openshift-default-dockercfg-dc8gp\"" Sep 14 02:05:07.097822 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.097792 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:05:07.116091 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116416 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116559 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116671 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116814 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplrg\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116951 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116951 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.116951 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.117221 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.116968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.118076 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.118035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:07.217481 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.217691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.217691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.217691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.217691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.217691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218004 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218176 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.217895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218176 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplrg\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218176 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218176 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218411 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218411 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.218519 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.218486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.220114 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.220096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.220207 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.220114 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.224962 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.224940 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.225124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.225103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplrg\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg\") pod \"e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.391180 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.391104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:07.394128 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.394055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:07.670735 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.670710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:05:07.672165 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:05:07.672137 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d5fe55_9c22_4aeb_af92_fdbe8c890d65.slice/crio-1a73ef4ff10020eb56145517177dcc3fad951dc418d3619afb3c70fa9bed5714 WatchSource:0}: Error finding container 1a73ef4ff10020eb56145517177dcc3fad951dc418d3619afb3c70fa9bed5714: Status 404 returned error can't find the container with id 1a73ef4ff10020eb56145517177dcc3fad951dc418d3619afb3c70fa9bed5714 Sep 14 02:05:07.674473 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.674442 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:07.674552 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.674507 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:07.674552 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:07.674535 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:08.220703 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:08.220660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerStarted","Data":"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5"} Sep 14 02:05:08.220902 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:08.220712 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerStarted","Data":"1a73ef4ff10020eb56145517177dcc3fad951dc418d3619afb3c70fa9bed5714"} Sep 14 02:05:08.241034 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:08.240983 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podStartSLOduration=1.24096731 podStartE2EDuration="1.24096731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:05:08.23890526 +0000 UTC m=+2053.472270542" watchObservedRunningTime="2026-09-14 02:05:08.24096731 +0000 UTC m=+2053.474332566" Sep 14 02:05:08.394993 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:08.394963 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:08.400028 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:08.400004 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:09.227109 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:09.227073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:09.228544 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:09.228524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:09.926495 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:09.926442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:10.162817 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:10.162779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:10.261000 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:10.260963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:10.919754 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:10.919725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:11.637528 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:11.637489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:12.242527 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:12.242494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:12.958685 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:12.958652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:13.060770 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:13.060716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:13.499897 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:13.499850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:13.590071 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:13.590021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:13.862107 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:13.861989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:29.360202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.360123 2579 generic.go:356] "Generic (PLEG): container finished" podID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerID="3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5" exitCode=0 Sep 14 02:05:29.360600 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.360199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerDied","Data":"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5"} Sep 14 02:05:29.360670 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.360655 2579 scope.go:117] "RemoveContainer" containerID="3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5" Sep 14 02:05:29.361762 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.361736 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:29.361826 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.361802 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:29.361864 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:29.361831 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 14 02:05:30.367289 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:30.367249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerStarted","Data":"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a"} Sep 14 02:05:30.394992 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:30.394966 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:30.395197 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:30.395109 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:30.395197 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:30.395171 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:31.285366 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:31.285332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:31.395380 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:31.395342 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:31.395825 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:31.395399 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:32.395278 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:32.395248 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:32.395448 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:32.395295 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:33.394757 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:33.394723 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:33.394995 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:33.394785 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:34.395205 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:34.395174 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:34.395674 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:34.395234 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:35.394438 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:35.394400 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:35.394613 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:35.394460 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:35.631846 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:35.631808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:36.344040 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:36.344002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:36.395427 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:36.395389 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Sep 14 02:05:36.395607 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:36.395445 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Sep 14 02:05:36.941660 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:36.941605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:37.398631 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:37.398604 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:37.398913 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:37.398879 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:37.399765 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:37.399743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:05:37.551330 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:37.551298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:38.308767 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:38.308732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:38.925449 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:38.925412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:39.220553 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:39.220462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:39.317857 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:39.317817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:40.300764 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:40.300732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:41.011691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.011658 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl"] Sep 14 02:05:41.015653 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.015632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:41.019002 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.018953 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 02:05:41.019455 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.019425 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:05:41.032662 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.032621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl"] Sep 14 02:05:41.060708 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.060671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl\" (UID: \"cc57c7af-cb6a-4181-8d96-4692b0525b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:41.161650 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.161613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl\" (UID: \"cc57c7af-cb6a-4181-8d96-4692b0525b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:41.162339 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.162313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl\" (UID: \"cc57c7af-cb6a-4181-8d96-4692b0525b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:41.336748 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.336652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:41.607824 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.607795 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl"] Sep 14 02:05:41.609447 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:05:41.609420 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc57c7af_cb6a_4181_8d96_4692b0525b36.slice/crio-1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c WatchSource:0}: Error finding container 1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c: Status 404 returned error can't find the container with id 1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c Sep 14 02:05:41.828250 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:41.828216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:42.344003 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:42.343970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:42.436105 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:42.436075 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc57c7af-cb6a-4181-8d96-4692b0525b36" containerID="117188a797ec571fdf9fc35b81adfbb0978185ea3faba38b2e8a837a75fbcf5b" exitCode=0 Sep 14 02:05:42.436299 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:42.436173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" event={"ID":"cc57c7af-cb6a-4181-8d96-4692b0525b36","Type":"ContainerDied","Data":"117188a797ec571fdf9fc35b81adfbb0978185ea3faba38b2e8a837a75fbcf5b"} Sep 14 02:05:42.436299 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:42.436241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" event={"ID":"cc57c7af-cb6a-4181-8d96-4692b0525b36","Type":"ContainerStarted","Data":"1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c"} Sep 14 02:05:43.034670 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.034636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:43.578698 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.578673 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:43.644691 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.644649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:43.683711 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.683677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca\") pod \"cc57c7af-cb6a-4181-8d96-4692b0525b36\" (UID: \"cc57c7af-cb6a-4181-8d96-4692b0525b36\") " Sep 14 02:05:43.684025 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.684002 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc57c7af-cb6a-4181-8d96-4692b0525b36" (UID: "cc57c7af-cb6a-4181-8d96-4692b0525b36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:05:43.785069 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:43.785016 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc57c7af-cb6a-4181-8d96-4692b0525b36-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:05:44.448947 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:44.448904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:44.450845 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:44.450814 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" Sep 14 02:05:44.450985 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:44.450844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl" event={"ID":"cc57c7af-cb6a-4181-8d96-4692b0525b36","Type":"ContainerDied","Data":"1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c"} Sep 14 02:05:44.450985 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:44.450891 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d3cfd55251f23732c93bd587e0b9ad1777b7394b407c3e1d54c33d8ca9e1d1c" Sep 14 02:05:45.116367 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:45.116334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:51.806123 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.806080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:51.878978 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.878926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:51.936320 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.936280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:51.984904 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.984865 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl"] Sep 14 02:05:51.987100 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.987080 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-072cc4-swpzl"] Sep 14 02:05:51.991217 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:51.991186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:52.408958 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:52.408919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:52.467799 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:52.467763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:53.157134 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:53.157101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:53.210288 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:53.210255 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc57c7af-cb6a-4181-8d96-4692b0525b36" path="/var/lib/kubelet/pods/cc57c7af-cb6a-4181-8d96-4692b0525b36/volumes" Sep 14 02:05:53.937194 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:53.937159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:54.650274 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:54.650236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:55.327179 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:55.327140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:55.359138 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:55.359109 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:05:55.361714 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:55.361687 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:05:55.756591 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:55.756560 2579 scope.go:117] "RemoveContainer" containerID="cef302248dfed8e54d24707950d740a98e99cd5bd88574842388af3951f4d855" Sep 14 02:05:55.767532 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:55.767505 2579 scope.go:117] "RemoveContainer" containerID="ae6bbd7a793994af14018b32731a41b583d07c7ef743b66cdcd97dda6857334d" Sep 14 02:05:58.260781 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:58.260739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:58.322616 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:58.322574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:05:58.616468 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:58.616373 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:05:58.616637 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:58.616576 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" containerID="cri-o://c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a" gracePeriod=30 Sep 14 02:05:58.640370 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:05:58.640339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:03.634947 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.634915 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd"] Sep 14 02:06:03.635415 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.635399 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc57c7af-cb6a-4181-8d96-4692b0525b36" containerName="revoke-keys" Sep 14 02:06:03.635454 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.635416 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc57c7af-cb6a-4181-8d96-4692b0525b36" containerName="revoke-keys" Sep 14 02:06:03.635512 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.635503 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc57c7af-cb6a-4181-8d96-4692b0525b36" containerName="revoke-keys" Sep 14 02:06:03.638608 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.638587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:03.641030 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.641004 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 02:06:03.641226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.641206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:06:03.645292 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.645271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd"] Sep 14 02:06:03.756005 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.755969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:03.802760 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.802724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd\" (UID: \"613af563-5f11-4dda-a3be-a6b1f4759e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:03.865287 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.865262 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:06:03.903841 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.903744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd\" (UID: \"613af563-5f11-4dda-a3be-a6b1f4759e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:03.904415 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:03.904389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd\" (UID: \"613af563-5f11-4dda-a3be-a6b1f4759e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:04.003388 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.003347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:04.004642 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004752 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004675 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004752 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004752 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004756 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004863 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gplrg\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.004928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.004909 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data\") pod \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\" (UID: \"04d5fe55-9c22-4aeb-af92-fdbe8c890d65\") " Sep 14 02:06:04.005232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005012 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:06:04.005232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005040 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:06:04.005232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:06:04.005232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005162 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:06:04.005449 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data" (OuterVolumeSpecName: "istio-data") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:06:04.005449 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005402 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.005449 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005429 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.005449 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005444 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.005644 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.005457 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.007391 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.007362 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg" (OuterVolumeSpecName: "kube-api-access-gplrg") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "kube-api-access-gplrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:06:04.007513 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.007448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:06:04.007513 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.007495 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:06:04.007629 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.007504 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token" (OuterVolumeSpecName: "istio-token") pod "04d5fe55-9c22-4aeb-af92-fdbe8c890d65" (UID: "04d5fe55-9c22-4aeb-af92-fdbe8c890d65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:06:04.106527 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.106487 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.106527 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.106529 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.106734 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.106544 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.106734 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.106558 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplrg\" (UniqueName: \"kubernetes.io/projected/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-kube-api-access-gplrg\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.106734 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.106572 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d5fe55-9c22-4aeb-af92-fdbe8c890d65-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:04.276382 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.276356 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd"] Sep 14 02:06:04.278108 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:06:04.278082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613af563_5f11_4dda_a3be_a6b1f4759e83.slice/crio-03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68 WatchSource:0}: Error finding container 03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68: Status 404 returned error can't find the container with id 03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68 Sep 14 02:06:04.468139 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.468103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:04.571793 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.571704 2579 generic.go:356] "Generic (PLEG): container finished" podID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerID="c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a" exitCode=0 Sep 14 02:06:04.571793 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.571750 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" Sep 14 02:06:04.572014 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.571790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerDied","Data":"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a"} Sep 14 02:06:04.572014 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.571847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84" event={"ID":"04d5fe55-9c22-4aeb-af92-fdbe8c890d65","Type":"ContainerDied","Data":"1a73ef4ff10020eb56145517177dcc3fad951dc418d3619afb3c70fa9bed5714"} Sep 14 02:06:04.572014 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.571871 2579 scope.go:117] "RemoveContainer" containerID="c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a" Sep 14 02:06:04.573168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.573146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" event={"ID":"613af563-5f11-4dda-a3be-a6b1f4759e83","Type":"ContainerStarted","Data":"03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68"} Sep 14 02:06:04.581489 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.581471 2579 scope.go:117] "RemoveContainer" containerID="3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.591401 2579 scope.go:117] "RemoveContainer" containerID="c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:06:04.591942 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a\": container with ID starting with c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a not found: ID does not exist" containerID="c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.591985 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a"} err="failed to get container status \"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a\": rpc error: code = NotFound desc = could not find container \"c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a\": container with ID starting with c52355b89164696345b34867d117a51068c7d0b6004d57ac9345851fffb6321a not found: ID does not exist" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.592011 2579 scope.go:117] "RemoveContainer" containerID="3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:06:04.593968 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5\": container with ID starting with 3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5 not found: ID does not exist" containerID="3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5" Sep 14 02:06:04.594019 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.593995 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5"} err="failed to get container status \"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5\": rpc error: code = NotFound desc = could not find container \"3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5\": container with ID starting with 3803157a020d6a474ffffe9d3a7639317709516718e52e3030a6425dfec529f5 not found: ID does not exist" Sep 14 02:06:04.596144 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.595908 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:06:04.598632 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:04.598608 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-072cc4-openshift-default-55fdf978f8-cjl84"] Sep 14 02:06:05.062456 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:05.062416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:05.211277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:05.211247 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" path="/var/lib/kubelet/pods/04d5fe55-9c22-4aeb-af92-fdbe8c890d65/volumes" Sep 14 02:06:05.581013 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:05.580974 2579 generic.go:356] "Generic (PLEG): container finished" podID="613af563-5f11-4dda-a3be-a6b1f4759e83" containerID="2783f8cf86df5e14066923bd2e5adf1dba95534c5e6770cf7e28ff305d724cfb" exitCode=0 Sep 14 02:06:05.581184 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:05.581075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" event={"ID":"613af563-5f11-4dda-a3be-a6b1f4759e83","Type":"ContainerDied","Data":"2783f8cf86df5e14066923bd2e5adf1dba95534c5e6770cf7e28ff305d724cfb"} Sep 14 02:06:05.673936 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:05.673895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:06.315049 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:06.315001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:06.721923 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:06.721892 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:06.832975 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:06.832943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca\") pod \"613af563-5f11-4dda-a3be-a6b1f4759e83\" (UID: \"613af563-5f11-4dda-a3be-a6b1f4759e83\") " Sep 14 02:06:06.833347 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:06.833324 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "613af563-5f11-4dda-a3be-a6b1f4759e83" (UID: "613af563-5f11-4dda-a3be-a6b1f4759e83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:06:06.934577 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:06.934498 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613af563-5f11-4dda-a3be-a6b1f4759e83-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:07.042754 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:07.042722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:07.593690 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:07.593661 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" Sep 14 02:06:07.594109 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:07.593695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd" event={"ID":"613af563-5f11-4dda-a3be-a6b1f4759e83","Type":"ContainerDied","Data":"03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68"} Sep 14 02:06:07.594109 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:07.593740 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03dd354f87a7e0a74e9521fcb3ea99bd065fd157fafa2d27832705f5692b9e68" Sep 14 02:06:19.069354 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.069318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:19.096154 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.096118 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:06:19.096408 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.096373 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" podUID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" containerName="payload-pre-processing" containerID="cri-o://27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d" gracePeriod=30 Sep 14 02:06:19.170099 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.170036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:19.238399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.238363 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd"] Sep 14 02:06:19.240086 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.240039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:19.240959 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.240938 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-39c298-5mkxd"] Sep 14 02:06:19.296471 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.296426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:19.364522 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.364496 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:06:19.452352 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.452313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume\") pod \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " Sep 14 02:06:19.452529 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.452460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4frj4\" (UniqueName: \"kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4\") pod \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\" (UID: \"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d\") " Sep 14 02:06:19.452718 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.452692 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" (UID: "1ae94c80-6a55-43ea-94b1-60c7aefa8d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:06:19.454521 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.454498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4" (OuterVolumeSpecName: "kube-api-access-4frj4") pod "1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" (UID: "1ae94c80-6a55-43ea-94b1-60c7aefa8d8d"). InnerVolumeSpecName "kube-api-access-4frj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:06:19.553355 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.553315 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frj4\" (UniqueName: \"kubernetes.io/projected/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-kube-api-access-4frj4\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.553355 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.553347 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.625391 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.625301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:19.658107 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.658072 2579 generic.go:356] "Generic (PLEG): container finished" podID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" containerID="27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d" exitCode=0 Sep 14 02:06:19.658312 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.658131 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" Sep 14 02:06:19.658312 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.658156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" event={"ID":"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d","Type":"ContainerDied","Data":"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d"} Sep 14 02:06:19.658312 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.658222 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf" event={"ID":"1ae94c80-6a55-43ea-94b1-60c7aefa8d8d","Type":"ContainerDied","Data":"1f8ac627a5a4addb906fa925cf7fce935436166fa69d2c6e16bf461a4e3ff411"} Sep 14 02:06:19.658312 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.658253 2579 scope.go:117] "RemoveContainer" containerID="27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d" Sep 14 02:06:19.667896 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.667878 2579 scope.go:117] "RemoveContainer" containerID="27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d" Sep 14 02:06:19.668234 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:06:19.668213 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d\": container with ID starting with 27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d not found: ID does not exist" containerID="27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d" Sep 14 02:06:19.668316 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.668241 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d"} err="failed to get container status \"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d\": rpc error: code = NotFound desc = could not find container \"27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d\": container with ID starting with 27876b3b48ced77dda1e9adf5a5925b522214f2f465622b1e518627dc8651c5d not found: ID does not exist" Sep 14 02:06:19.679737 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.679710 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:06:19.684629 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.684604 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-39c298-844858875d-c6crf"] Sep 14 02:06:19.686183 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:19.686152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:20.425160 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:20.425121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:21.038808 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:21.038773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:21.210835 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:21.210797 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" path="/var/lib/kubelet/pods/1ae94c80-6a55-43ea-94b1-60c7aefa8d8d/volumes" Sep 14 02:06:21.211345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:21.211326 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613af563-5f11-4dda-a3be-a6b1f4759e83" path="/var/lib/kubelet/pods/613af563-5f11-4dda-a3be-a6b1f4759e83/volumes" Sep 14 02:06:21.656416 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:21.656373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:24.649470 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:24.649418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:06:55.794611 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:55.794527 2579 scope.go:117] "RemoveContainer" containerID="93aa475bbc9cd802b82ee39f54cea83918b88e1be6b5c95209eaa4c4e4e38ec3" Sep 14 02:06:55.803248 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:06:55.803230 2579 scope.go:117] "RemoveContainer" containerID="96c148ff2bbdb3fc49abf6b237c338e8df00279c9390dd8e9a4444ffd1dedaa8" Sep 14 02:07:01.007627 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:01.007592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:02.502111 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:02.502074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:07.063629 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.063594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:07.123936 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.123892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:07.324134 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.324031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:07.680168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680088 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m"] Sep 14 02:07:07.680559 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680546 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680603 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680561 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680603 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680577 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680603 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680582 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680603 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680593 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613af563-5f11-4dda-a3be-a6b1f4759e83" containerName="revoke-keys" Sep 14 02:07:07.680603 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680602 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="613af563-5f11-4dda-a3be-a6b1f4759e83" containerName="revoke-keys" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680612 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" containerName="payload-pre-processing" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680618 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" containerName="payload-pre-processing" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680690 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae94c80-6a55-43ea-94b1-60c7aefa8d8d" containerName="payload-pre-processing" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680698 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680707 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d5fe55-9c22-4aeb-af92-fdbe8c890d65" containerName="istio-proxy" Sep 14 02:07:07.680753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.680715 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="613af563-5f11-4dda-a3be-a6b1f4759e83" containerName="revoke-keys" Sep 14 02:07:07.683876 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.683860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:07.686432 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.686411 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:07:07.686523 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.686411 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 02:07:07.690995 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.690971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m"] Sep 14 02:07:07.704323 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.704290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m\" (UID: \"16ef7bf4-abc1-4960-9595-fbcbd7d465bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:07.805414 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.805381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m\" (UID: \"16ef7bf4-abc1-4960-9595-fbcbd7d465bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:07.806019 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.805998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m\" (UID: \"16ef7bf4-abc1-4960-9595-fbcbd7d465bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:07.835213 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.835183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:07.995202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:07.995162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:08.265393 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:08.265370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m"] Sep 14 02:07:08.267684 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:07:08.267657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ef7bf4_abc1_4960_9595_fbcbd7d465bb.slice/crio-fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069 WatchSource:0}: Error finding container fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069: Status 404 returned error can't find the container with id fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069 Sep 14 02:07:08.623724 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:08.623694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:08.941669 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:08.941569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" event={"ID":"16ef7bf4-abc1-4960-9595-fbcbd7d465bb","Type":"ContainerStarted","Data":"a93db4b528c39f746bfa1718c53451d2b8fd12a0532f628cf2b35e4f0afcfd10"} Sep 14 02:07:08.941669 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:08.941634 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" event={"ID":"16ef7bf4-abc1-4960-9595-fbcbd7d465bb","Type":"ContainerStarted","Data":"fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069"} Sep 14 02:07:08.961402 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:08.961349 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" podStartSLOduration=1.96133377 podStartE2EDuration="1.96133377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:07:08.960456707 +0000 UTC m=+2174.193821964" watchObservedRunningTime="2026-09-14 02:07:08.96133377 +0000 UTC m=+2174.194699026" Sep 14 02:07:09.328144 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:09.328112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:09.945933 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:09.945893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:09.949952 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:09.949908 2579 generic.go:356] "Generic (PLEG): container finished" podID="16ef7bf4-abc1-4960-9595-fbcbd7d465bb" containerID="a93db4b528c39f746bfa1718c53451d2b8fd12a0532f628cf2b35e4f0afcfd10" exitCode=0 Sep 14 02:07:09.950134 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:09.949949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" event={"ID":"16ef7bf4-abc1-4960-9595-fbcbd7d465bb","Type":"ContainerDied","Data":"a93db4b528c39f746bfa1718c53451d2b8fd12a0532f628cf2b35e4f0afcfd10"} Sep 14 02:07:10.628499 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:10.628464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:11.090214 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.090186 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:11.136933 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.136904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca\") pod \"16ef7bf4-abc1-4960-9595-fbcbd7d465bb\" (UID: \"16ef7bf4-abc1-4960-9595-fbcbd7d465bb\") " Sep 14 02:07:11.137311 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.137288 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16ef7bf4-abc1-4960-9595-fbcbd7d465bb" (UID: "16ef7bf4-abc1-4960-9595-fbcbd7d465bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:11.237713 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.237688 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef7bf4-abc1-4960-9595-fbcbd7d465bb-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:11.963271 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.963241 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" Sep 14 02:07:11.963677 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.963273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m" event={"ID":"16ef7bf4-abc1-4960-9595-fbcbd7d465bb","Type":"ContainerDied","Data":"fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069"} Sep 14 02:07:11.963677 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:11.963328 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb17ee555c0464a2b76562d0c223a1f23d75ee15f2fb1f8abd7504420432b069" Sep 14 02:07:18.168998 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.168962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.191459 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.191427 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:07:18.191677 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.191649 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" podUID="26634479-ceea-44e8-9471-69fffb0c337d" containerName="payload-processing" containerID="cri-o://dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3" gracePeriod=30 Sep 14 02:07:18.197396 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.197311 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:07:18.197733 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.197707 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" podUID="64225315-143b-42c1-8d55-642270d7b2d7" containerName="payload-pre-processing" containerID="cri-o://10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6" gracePeriod=30 Sep 14 02:07:18.229232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.229196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.284927 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.284889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.342648 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.341937 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m"] Sep 14 02:07:18.343307 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.343277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.345249 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.345223 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8020f1-47d2m"] Sep 14 02:07:18.399413 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.399377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.490149 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.490125 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:07:18.492738 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.492718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:07:18.623917 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.623884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6spcj\" (UniqueName: \"kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj\") pod \"64225315-143b-42c1-8d55-642270d7b2d7\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " Sep 14 02:07:18.624145 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.623936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume\") pod \"64225315-143b-42c1-8d55-642270d7b2d7\" (UID: \"64225315-143b-42c1-8d55-642270d7b2d7\") " Sep 14 02:07:18.624145 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.623970 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume\") pod \"26634479-ceea-44e8-9471-69fffb0c337d\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " Sep 14 02:07:18.624145 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.624050 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhgxj\" (UniqueName: \"kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj\") pod \"26634479-ceea-44e8-9471-69fffb0c337d\" (UID: \"26634479-ceea-44e8-9471-69fffb0c337d\") " Sep 14 02:07:18.624462 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.624426 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64225315-143b-42c1-8d55-642270d7b2d7" (UID: "64225315-143b-42c1-8d55-642270d7b2d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:18.624462 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.624434 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26634479-ceea-44e8-9471-69fffb0c337d" (UID: "26634479-ceea-44e8-9471-69fffb0c337d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:18.626248 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.626225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj" (OuterVolumeSpecName: "kube-api-access-6spcj") pod "64225315-143b-42c1-8d55-642270d7b2d7" (UID: "64225315-143b-42c1-8d55-642270d7b2d7"). InnerVolumeSpecName "kube-api-access-6spcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:18.626337 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.626267 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj" (OuterVolumeSpecName: "kube-api-access-qhgxj") pod "26634479-ceea-44e8-9471-69fffb0c337d" (UID: "26634479-ceea-44e8-9471-69fffb0c337d"). InnerVolumeSpecName "kube-api-access-qhgxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:18.725120 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.725020 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgxj\" (UniqueName: \"kubernetes.io/projected/26634479-ceea-44e8-9471-69fffb0c337d-kube-api-access-qhgxj\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:18.725120 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.725048 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6spcj\" (UniqueName: \"kubernetes.io/projected/64225315-143b-42c1-8d55-642270d7b2d7-kube-api-access-6spcj\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:18.725120 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.725068 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64225315-143b-42c1-8d55-642270d7b2d7-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:18.725120 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.725077 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26634479-ceea-44e8-9471-69fffb0c337d-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:18.753488 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.753452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:18.854538 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:18.854500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:19.004259 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.004225 2579 generic.go:356] "Generic (PLEG): container finished" podID="26634479-ceea-44e8-9471-69fffb0c337d" containerID="dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3" exitCode=0 Sep 14 02:07:19.004436 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.004277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" Sep 14 02:07:19.004436 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.004305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" event={"ID":"26634479-ceea-44e8-9471-69fffb0c337d","Type":"ContainerDied","Data":"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3"} Sep 14 02:07:19.004436 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.004353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj" event={"ID":"26634479-ceea-44e8-9471-69fffb0c337d","Type":"ContainerDied","Data":"988d460211a1c9a49f6fb541ed62d070ac145c470920f8dd61affd1dc2a7f096"} Sep 14 02:07:19.004436 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.004380 2579 scope.go:117] "RemoveContainer" containerID="dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3" Sep 14 02:07:19.005850 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.005829 2579 generic.go:356] "Generic (PLEG): container finished" podID="64225315-143b-42c1-8d55-642270d7b2d7" containerID="10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6" exitCode=0 Sep 14 02:07:19.005994 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.005874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" event={"ID":"64225315-143b-42c1-8d55-642270d7b2d7","Type":"ContainerDied","Data":"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6"} Sep 14 02:07:19.005994 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.005873 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" Sep 14 02:07:19.005994 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.005910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9" event={"ID":"64225315-143b-42c1-8d55-642270d7b2d7","Type":"ContainerDied","Data":"75d0161d489cee07ac55eb7438b493efc714f64a2a0dfe87eda2b5235708a543"} Sep 14 02:07:19.018216 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.018197 2579 scope.go:117] "RemoveContainer" containerID="dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3" Sep 14 02:07:19.018550 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:19.018524 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3\": container with ID starting with dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3 not found: ID does not exist" containerID="dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3" Sep 14 02:07:19.018636 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.018556 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3"} err="failed to get container status \"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3\": rpc error: code = NotFound desc = could not find container \"dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3\": container with ID starting with dbea2e60fcf9b4e337f41476790b93ca080d1a15a0892ed743861593f14714c3 not found: ID does not exist" Sep 14 02:07:19.018636 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.018574 2579 scope.go:117] "RemoveContainer" containerID="10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6" Sep 14 02:07:19.027183 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.027158 2579 scope.go:117] "RemoveContainer" containerID="10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6" Sep 14 02:07:19.027453 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:19.027432 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6\": container with ID starting with 10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6 not found: ID does not exist" containerID="10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6" Sep 14 02:07:19.027516 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.027462 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6"} err="failed to get container status \"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6\": rpc error: code = NotFound desc = could not find container \"10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6\": container with ID starting with 10e5118f1790f431865efdbe4ccf99e8a832cc7e9916aed31a2aa16a400803f6 not found: ID does not exist" Sep 14 02:07:19.031728 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.031701 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:07:19.037314 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.037291 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8020f1-84b75ddc97-bv8bj"] Sep 14 02:07:19.047295 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.047271 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:07:19.052722 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.052699 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8020f1-64c94dd578-9c8x9"] Sep 14 02:07:19.210901 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.210863 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ef7bf4-abc1-4960-9595-fbcbd7d465bb" path="/var/lib/kubelet/pods/16ef7bf4-abc1-4960-9595-fbcbd7d465bb/volumes" Sep 14 02:07:19.211353 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.211246 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26634479-ceea-44e8-9471-69fffb0c337d" path="/var/lib/kubelet/pods/26634479-ceea-44e8-9471-69fffb0c337d/volumes" Sep 14 02:07:19.211577 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.211563 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64225315-143b-42c1-8d55-642270d7b2d7" path="/var/lib/kubelet/pods/64225315-143b-42c1-8d55-642270d7b2d7/volumes" Sep 14 02:07:19.396821 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:19.396721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:20.032871 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:20.032823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:24.308916 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.308871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:24.408928 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.408885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:24.559513 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.559429 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:07:24.559714 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.559687 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" containerID="cri-o://505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8" gracePeriod=30 Sep 14 02:07:24.582738 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.582706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:24.910484 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.910401 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp"] Sep 14 02:07:24.911024 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911006 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26634479-ceea-44e8-9471-69fffb0c337d" containerName="payload-processing" Sep 14 02:07:24.911093 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911030 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26634479-ceea-44e8-9471-69fffb0c337d" containerName="payload-processing" Sep 14 02:07:24.911093 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911073 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ef7bf4-abc1-4960-9595-fbcbd7d465bb" containerName="revoke-keys" Sep 14 02:07:24.911093 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911082 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ef7bf4-abc1-4960-9595-fbcbd7d465bb" containerName="revoke-keys" Sep 14 02:07:24.911202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911101 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64225315-143b-42c1-8d55-642270d7b2d7" containerName="payload-pre-processing" Sep 14 02:07:24.911202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911111 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64225315-143b-42c1-8d55-642270d7b2d7" containerName="payload-pre-processing" Sep 14 02:07:24.911260 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911218 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ef7bf4-abc1-4960-9595-fbcbd7d465bb" containerName="revoke-keys" Sep 14 02:07:24.911260 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911231 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26634479-ceea-44e8-9471-69fffb0c337d" containerName="payload-processing" Sep 14 02:07:24.911260 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.911240 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64225315-143b-42c1-8d55-642270d7b2d7" containerName="payload-pre-processing" Sep 14 02:07:24.916504 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.916477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:24.919509 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.919475 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp"] Sep 14 02:07:24.919649 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.919516 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 02:07:24.919649 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.919537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:07:24.984532 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:24.984494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp\" (UID: \"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:25.085226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.085193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp\" (UID: \"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:25.085860 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.085838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp\" (UID: \"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:25.229910 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.229886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:25.241654 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.241624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:25.500564 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.500532 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp"] Sep 14 02:07:25.502178 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:07:25.502140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07e34b0_3098_4f7d_8458_ab0e1e1f9e21.slice/crio-3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6 WatchSource:0}: Error finding container 3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6: Status 404 returned error can't find the container with id 3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6 Sep 14 02:07:25.929778 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:25.929744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:26.051600 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:26.051565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" event={"ID":"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21","Type":"ContainerStarted","Data":"6b12223a8803e5233449e27d1be8391eb87f40c360547e3f693268868a50a3f3"} Sep 14 02:07:26.051774 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:26.051640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" event={"ID":"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21","Type":"ContainerStarted","Data":"3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6"} Sep 14 02:07:26.630136 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:26.630034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:27.058510 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:27.058475 2579 generic.go:356] "Generic (PLEG): container finished" podID="f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" containerID="6b12223a8803e5233449e27d1be8391eb87f40c360547e3f693268868a50a3f3" exitCode=0 Sep 14 02:07:27.058658 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:27.058567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" event={"ID":"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21","Type":"ContainerDied","Data":"6b12223a8803e5233449e27d1be8391eb87f40c360547e3f693268868a50a3f3"} Sep 14 02:07:27.334043 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:27.333951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:28.199661 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.199633 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:28.321752 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.321653 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca\") pod \"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21\" (UID: \"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21\") " Sep 14 02:07:28.322028 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.322002 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" (UID: "f07e34b0-3098-4f7d-8458-ab0e1e1f9e21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:28.423053 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.423015 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:28.707571 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.707488 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 02:07:28.707711 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:28.707570 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 02:07:29.071124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.071088 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" Sep 14 02:07:29.071124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.071106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp" event={"ID":"f07e34b0-3098-4f7d-8458-ab0e1e1f9e21","Type":"ContainerDied","Data":"3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6"} Sep 14 02:07:29.071359 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.071154 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f076ef9c4b037b94f8d7a4e3fdbf357ac8575ca4a7e98eefefac7fc0c5f38c6" Sep 14 02:07:29.812566 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.812542 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:07:29.936594 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936505 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr2sd\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936594 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936594 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936880 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936607 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936880 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936639 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936880 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936660 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936880 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936690 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.936880 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.937168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936910 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket\") pod \"04ec6501-01d3-4ea2-894e-399f6873534c\" (UID: \"04ec6501-01d3-4ea2-894e-399f6873534c\") " Sep 14 02:07:29.937168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.936934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data" (OuterVolumeSpecName: "istio-data") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:29.937168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937015 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:29.937168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:29.937376 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937242 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:29.937376 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937358 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:29.937376 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937368 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:29.937526 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937381 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:29.937526 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937395 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:29.937526 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.937408 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:29.938859 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.938830 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:07:29.938971 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.938881 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd" (OuterVolumeSpecName: "kube-api-access-vr2sd") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "kube-api-access-vr2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:29.939306 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.939284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:29.939359 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:29.939300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token" (OuterVolumeSpecName: "istio-token") pod "04ec6501-01d3-4ea2-894e-399f6873534c" (UID: "04ec6501-01d3-4ea2-894e-399f6873534c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:30.038840 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.038803 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:30.038840 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.038834 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04ec6501-01d3-4ea2-894e-399f6873534c-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:30.038840 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.038842 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04ec6501-01d3-4ea2-894e-399f6873534c-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:30.039094 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.038850 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr2sd\" (UniqueName: \"kubernetes.io/projected/04ec6501-01d3-4ea2-894e-399f6873534c-kube-api-access-vr2sd\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:30.039094 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.038861 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04ec6501-01d3-4ea2-894e-399f6873534c-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:30.079051 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.079019 2579 generic.go:356] "Generic (PLEG): container finished" podID="04ec6501-01d3-4ea2-894e-399f6873534c" containerID="505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8" exitCode=0 Sep 14 02:07:30.079266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.079077 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" Sep 14 02:07:30.079266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.079101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerDied","Data":"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8"} Sep 14 02:07:30.079266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.079159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b" event={"ID":"04ec6501-01d3-4ea2-894e-399f6873534c","Type":"ContainerDied","Data":"dbc714c4e8e97a27ec05502c65dae236819d13c5e6b7dc7e9a974cbea3360508"} Sep 14 02:07:30.079266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.079187 2579 scope.go:117] "RemoveContainer" containerID="505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8" Sep 14 02:07:30.091721 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.091699 2579 scope.go:117] "RemoveContainer" containerID="38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea" Sep 14 02:07:30.100189 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.100168 2579 scope.go:117] "RemoveContainer" containerID="505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8" Sep 14 02:07:30.100456 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:30.100432 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8\": container with ID starting with 505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8 not found: ID does not exist" containerID="505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8" Sep 14 02:07:30.100529 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.100461 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8"} err="failed to get container status \"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8\": rpc error: code = NotFound desc = could not find container \"505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8\": container with ID starting with 505ec5688f2cee9491d9a57ef49dd575cf0d5a35bc93c0bdfbf328dd6ec90bd8 not found: ID does not exist" Sep 14 02:07:30.100529 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.100484 2579 scope.go:117] "RemoveContainer" containerID="38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea" Sep 14 02:07:30.100825 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:30.100793 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea\": container with ID starting with 38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea not found: ID does not exist" containerID="38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea" Sep 14 02:07:30.100921 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.100825 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea"} err="failed to get container status \"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea\": rpc error: code = NotFound desc = could not find container \"38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea\": container with ID starting with 38fe1a6368e99cb68f08c9185ab158bab27fd4fb4530947b0e0d812c8db11eea not found: ID does not exist" Sep 14 02:07:30.101687 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.101669 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:07:30.103699 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:30.103680 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8020f1-openshift-default-588fbf84f4-5b98b"] Sep 14 02:07:31.209964 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:31.209931 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" path="/var/lib/kubelet/pods/04ec6501-01d3-4ea2-894e-399f6873534c/volumes" Sep 14 02:07:35.313279 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.313244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:35.337433 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.337399 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 02:07:35.337663 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.337638 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" podUID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" containerName="payload-processing" containerID="cri-o://1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775" gracePeriod=30 Sep 14 02:07:35.343377 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.343348 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 02:07:35.343686 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.343660 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" podUID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" containerName="payload-pre-processing" containerID="cri-o://d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83" gracePeriod=30 Sep 14 02:07:35.375771 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.375728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:35.436258 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.436223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:35.479858 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.479816 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp"] Sep 14 02:07:35.484637 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.484607 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5d3ac7-k5vlp"] Sep 14 02:07:35.497051 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.497015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:35.631224 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.631199 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 02:07:35.634018 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.633996 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 02:07:35.691583 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.691555 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume\") pod \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " Sep 14 02:07:35.691749 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.691589 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume\") pod \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " Sep 14 02:07:35.691749 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.691652 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n67r8\" (UniqueName: \"kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8\") pod \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\" (UID: \"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d\") " Sep 14 02:07:35.691749 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.691676 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtrs\" (UniqueName: \"kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs\") pod \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\" (UID: \"bdaab2b9-e7ce-48bd-918e-683008ff22eb\") " Sep 14 02:07:35.692046 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.692018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" (UID: "1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:35.692158 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.692018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdaab2b9-e7ce-48bd-918e-683008ff22eb" (UID: "bdaab2b9-e7ce-48bd-918e-683008ff22eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:35.693683 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.693662 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8" (OuterVolumeSpecName: "kube-api-access-n67r8") pod "1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" (UID: "1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d"). InnerVolumeSpecName "kube-api-access-n67r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:35.693826 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.693804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs" (OuterVolumeSpecName: "kube-api-access-9mtrs") pod "bdaab2b9-e7ce-48bd-918e-683008ff22eb" (UID: "bdaab2b9-e7ce-48bd-918e-683008ff22eb"). InnerVolumeSpecName "kube-api-access-9mtrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:35.792399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.792355 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdaab2b9-e7ce-48bd-918e-683008ff22eb-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:35.792399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.792393 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:35.792399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.792406 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n67r8\" (UniqueName: \"kubernetes.io/projected/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d-kube-api-access-n67r8\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:35.792634 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.792418 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtrs\" (UniqueName: \"kubernetes.io/projected/bdaab2b9-e7ce-48bd-918e-683008ff22eb-kube-api-access-9mtrs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:35.839030 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.838998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:35.939153 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:35.939052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:36.113086 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.113027 2579 generic.go:356] "Generic (PLEG): container finished" podID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" containerID="1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775" exitCode=0 Sep 14 02:07:36.113086 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.113082 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" Sep 14 02:07:36.113311 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.113112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" event={"ID":"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d","Type":"ContainerDied","Data":"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775"} Sep 14 02:07:36.113311 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.113169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7" event={"ID":"1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d","Type":"ContainerDied","Data":"1025073c24c274d87af926f724bb37bd8ada5b506711fbf8ef608f1b8f73443c"} Sep 14 02:07:36.113311 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.113193 2579 scope.go:117] "RemoveContainer" containerID="1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775" Sep 14 02:07:36.114565 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.114541 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" containerID="d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83" exitCode=0 Sep 14 02:07:36.114657 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.114569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" event={"ID":"bdaab2b9-e7ce-48bd-918e-683008ff22eb","Type":"ContainerDied","Data":"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83"} Sep 14 02:07:36.114657 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.114594 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" Sep 14 02:07:36.114657 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.114620 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m" event={"ID":"bdaab2b9-e7ce-48bd-918e-683008ff22eb","Type":"ContainerDied","Data":"ce73ecbc0348760e393072905fce8e3f207f899bce5449e3e544bf3f10d5b0d0"} Sep 14 02:07:36.122719 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.122698 2579 scope.go:117] "RemoveContainer" containerID="1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775" Sep 14 02:07:36.123118 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:36.123004 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775\": container with ID starting with 1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775 not found: ID does not exist" containerID="1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775" Sep 14 02:07:36.123118 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.123038 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775"} err="failed to get container status \"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775\": rpc error: code = NotFound desc = could not find container \"1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775\": container with ID starting with 1f22d0df45d515fb6853cee021cb804d8e946fcbb6434d77ad1ad0c261f14775 not found: ID does not exist" Sep 14 02:07:36.123118 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.123084 2579 scope.go:117] "RemoveContainer" containerID="d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83" Sep 14 02:07:36.131593 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.131575 2579 scope.go:117] "RemoveContainer" containerID="d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83" Sep 14 02:07:36.131880 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:36.131864 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83\": container with ID starting with d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83 not found: ID does not exist" containerID="d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83" Sep 14 02:07:36.131929 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.131885 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83"} err="failed to get container status \"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83\": rpc error: code = NotFound desc = could not find container \"d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83\": container with ID starting with d97ede9fc13fe731cd8c891e54cabe41ab06e92aae3b6f223db1753e0df71d83 not found: ID does not exist" Sep 14 02:07:36.140337 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.140315 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 02:07:36.142251 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.142231 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5d3ac7-8698987bdf-25lt7"] Sep 14 02:07:36.150124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.150105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 02:07:36.153133 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.153113 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5d3ac7-f44c99d8b-64j4m"] Sep 14 02:07:36.732775 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:36.732741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:37.210753 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:37.210664 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" path="/var/lib/kubelet/pods/1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d/volumes" Sep 14 02:07:37.211154 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:37.211134 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" path="/var/lib/kubelet/pods/bdaab2b9-e7ce-48bd-918e-683008ff22eb/volumes" Sep 14 02:07:37.211502 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:37.211484 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" path="/var/lib/kubelet/pods/f07e34b0-3098-4f7d-8458-ab0e1e1f9e21/volumes" Sep 14 02:07:40.587741 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:40.587704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:41.557029 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:41.556994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:41.656794 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:41.656751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:41.821024 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:41.820939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:42.236750 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.236717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd"] Sep 14 02:07:42.237152 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237139 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:07:42.237204 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237156 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:07:42.237204 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237182 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:07:42.237204 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237188 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:07:42.237204 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237193 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" containerName="payload-pre-processing" Sep 14 02:07:42.237204 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237199 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" containerName="payload-pre-processing" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237210 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" containerName="revoke-keys" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237215 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" containerName="revoke-keys" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237229 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" containerName="payload-processing" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237238 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" containerName="payload-processing" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237301 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237312 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdaab2b9-e7ce-48bd-918e-683008ff22eb" containerName="payload-pre-processing" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237323 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1534c3a7-8dcc-4d2c-b18d-5c5bc3b9e73d" containerName="payload-processing" Sep 14 02:07:42.237364 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.237329 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07e34b0-3098-4f7d-8458-ab0e1e1f9e21" containerName="revoke-keys" Sep 14 02:07:42.242046 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.242018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:42.244986 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.244965 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dcfmz\"" Sep 14 02:07:42.245290 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.245273 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 02:07:42.258365 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.258342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd"] Sep 14 02:07:42.348744 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.348706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd\" (UID: \"ac6c78f2-369f-44f9-8f98-f7377ec70405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:42.449886 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.449854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd\" (UID: \"ac6c78f2-369f-44f9-8f98-f7377ec70405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:42.450492 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.450471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd\" (UID: \"ac6c78f2-369f-44f9-8f98-f7377ec70405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:42.552193 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.552109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:42.822586 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.822559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd"] Sep 14 02:07:42.824949 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:07:42.824912 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c78f2_369f_44f9_8f98_f7377ec70405.slice/crio-bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b WatchSource:0}: Error finding container bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b: Status 404 returned error can't find the container with id bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b Sep 14 02:07:42.906149 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:42.906092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:43.155853 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:43.155815 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" event={"ID":"ac6c78f2-369f-44f9-8f98-f7377ec70405","Type":"ContainerStarted","Data":"0589a203bac23a1f3f6b8bbc3ed1c747542bb177f968e65d56906316965a98bd"} Sep 14 02:07:43.155972 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:43.155912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" event={"ID":"ac6c78f2-369f-44f9-8f98-f7377ec70405","Type":"ContainerStarted","Data":"bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b"} Sep 14 02:07:43.171350 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:43.171302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" podStartSLOduration=1.171287846 podStartE2EDuration="1.171287846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:07:43.169363566 +0000 UTC m=+2208.402728863" watchObservedRunningTime="2026-09-14 02:07:43.171287846 +0000 UTC m=+2208.404653104" Sep 14 02:07:43.541824 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:43.541792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:44.163416 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:44.163386 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac6c78f2-369f-44f9-8f98-f7377ec70405" containerID="0589a203bac23a1f3f6b8bbc3ed1c747542bb177f968e65d56906316965a98bd" exitCode=0 Sep 14 02:07:44.163773 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:44.163474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" event={"ID":"ac6c78f2-369f-44f9-8f98-f7377ec70405","Type":"ContainerDied","Data":"0589a203bac23a1f3f6b8bbc3ed1c747542bb177f968e65d56906316965a98bd"} Sep 14 02:07:45.301450 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:45.301424 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:45.374904 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:45.374868 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca\") pod \"ac6c78f2-369f-44f9-8f98-f7377ec70405\" (UID: \"ac6c78f2-369f-44f9-8f98-f7377ec70405\") " Sep 14 02:07:45.375225 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:45.375204 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac6c78f2-369f-44f9-8f98-f7377ec70405" (UID: "ac6c78f2-369f-44f9-8f98-f7377ec70405"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:45.476108 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:45.476016 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c78f2-369f-44f9-8f98-f7377ec70405-maas-api-service-ca\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:46.175815 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:46.175778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" event={"ID":"ac6c78f2-369f-44f9-8f98-f7377ec70405","Type":"ContainerDied","Data":"bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b"} Sep 14 02:07:46.175815 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:46.175813 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd" Sep 14 02:07:46.176102 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:46.175827 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf3a4f880068d51caf373fa85900cbe6d90f5b68f998328f38947a3923d928b" Sep 14 02:07:52.800866 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.800828 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd"] Sep 14 02:07:52.802941 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.802903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:52.804646 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.804621 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-86995c-29822520-mp7kd"] Sep 14 02:07:52.827652 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.827623 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 02:07:52.827993 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.827939 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" podUID="ec437c4f-9b5b-43e8-ae08-7352471debde" containerName="payload-processing" containerID="cri-o://b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a" gracePeriod=30 Sep 14 02:07:52.838800 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.838777 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 02:07:52.839002 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.838966 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" podUID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" containerName="payload-pre-processing" containerID="cri-o://e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6" gracePeriod=30 Sep 14 02:07:52.908974 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:52.908945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.008924 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.008886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.088122 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.087997 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd"] Sep 14 02:07:53.090350 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.090259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.090684 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.090662 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-86995c-ntbcd"] Sep 14 02:07:53.117141 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.117119 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 02:07:53.120138 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.120119 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 02:07:53.148358 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.148333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlg2h\" (UniqueName: \"kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h\") pod \"ec437c4f-9b5b-43e8-ae08-7352471debde\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " Sep 14 02:07:53.148464 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.148416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume\") pod \"ec437c4f-9b5b-43e8-ae08-7352471debde\" (UID: \"ec437c4f-9b5b-43e8-ae08-7352471debde\") " Sep 14 02:07:53.148848 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.148819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec437c4f-9b5b-43e8-ae08-7352471debde" (UID: "ec437c4f-9b5b-43e8-ae08-7352471debde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:53.150351 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.150328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h" (OuterVolumeSpecName: "kube-api-access-hlg2h") pod "ec437c4f-9b5b-43e8-ae08-7352471debde" (UID: "ec437c4f-9b5b-43e8-ae08-7352471debde"). InnerVolumeSpecName "kube-api-access-hlg2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:53.164255 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.162470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.210469 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.210434 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228fd95f-16f2-406d-978e-4db468d99355" path="/var/lib/kubelet/pods/228fd95f-16f2-406d-978e-4db468d99355/volumes" Sep 14 02:07:53.210968 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.210950 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6c78f2-369f-44f9-8f98-f7377ec70405" path="/var/lib/kubelet/pods/ac6c78f2-369f-44f9-8f98-f7377ec70405/volumes" Sep 14 02:07:53.216381 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.216356 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec437c4f-9b5b-43e8-ae08-7352471debde" containerID="b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a" exitCode=0 Sep 14 02:07:53.216505 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.216390 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" Sep 14 02:07:53.216505 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.216436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" event={"ID":"ec437c4f-9b5b-43e8-ae08-7352471debde","Type":"ContainerDied","Data":"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a"} Sep 14 02:07:53.216505 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.216493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl" event={"ID":"ec437c4f-9b5b-43e8-ae08-7352471debde","Type":"ContainerDied","Data":"9fc882a1f0c0a39a9ae3a7b98e93ad9db9e1949f04e4fd6c7a4dbd2f52d3e804"} Sep 14 02:07:53.216679 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.216518 2579 scope.go:117] "RemoveContainer" containerID="b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a" Sep 14 02:07:53.218286 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.218267 2579 generic.go:356] "Generic (PLEG): container finished" podID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" containerID="e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6" exitCode=0 Sep 14 02:07:53.218399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.218300 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" Sep 14 02:07:53.218399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.218324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" event={"ID":"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93","Type":"ContainerDied","Data":"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6"} Sep 14 02:07:53.218399 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.218360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s" event={"ID":"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93","Type":"ContainerDied","Data":"b53c71026a3d67110361b737b00e7424d919d510d5b1526b9a52e2c4d5e6d778"} Sep 14 02:07:53.227381 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.227366 2579 scope.go:117] "RemoveContainer" containerID="b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a" Sep 14 02:07:53.227631 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:53.227612 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a\": container with ID starting with b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a not found: ID does not exist" containerID="b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a" Sep 14 02:07:53.227696 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.227637 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a"} err="failed to get container status \"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a\": rpc error: code = NotFound desc = could not find container \"b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a\": container with ID starting with b03c433c4f0691a2ba9b9c2bd94ff33176ff962abf71544ea0ee3cd08c18462a not found: ID does not exist" Sep 14 02:07:53.227696 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.227654 2579 scope.go:117] "RemoveContainer" containerID="e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6" Sep 14 02:07:53.240967 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.240947 2579 scope.go:117] "RemoveContainer" containerID="e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6" Sep 14 02:07:53.241285 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:07:53.241262 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6\": container with ID starting with e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6 not found: ID does not exist" containerID="e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6" Sep 14 02:07:53.241365 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.241289 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6"} err="failed to get container status \"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6\": rpc error: code = NotFound desc = could not find container \"e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6\": container with ID starting with e23a505b485bf8491d643eb0a14b36fdf53cec4d3cfed605fe0645cd26631dd6 not found: ID does not exist" Sep 14 02:07:53.242197 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.242176 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 02:07:53.247904 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.247882 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-86995c-bbf4dd68f-8xdrl"] Sep 14 02:07:53.249778 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.249761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8nxb\" (UniqueName: \"kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb\") pod \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " Sep 14 02:07:53.249842 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.249828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume\") pod \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\" (UID: \"b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93\") " Sep 14 02:07:53.250206 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.250184 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlg2h\" (UniqueName: \"kubernetes.io/projected/ec437c4f-9b5b-43e8-ae08-7352471debde-kube-api-access-hlg2h\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.250286 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.250208 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec437c4f-9b5b-43e8-ae08-7352471debde-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.250286 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.250202 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" (UID: "b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:53.251822 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.251803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb" (OuterVolumeSpecName: "kube-api-access-n8nxb") pod "b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" (UID: "b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93"). InnerVolumeSpecName "kube-api-access-n8nxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:53.351588 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.351500 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nxb\" (UniqueName: \"kubernetes.io/projected/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-kube-api-access-n8nxb\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.351588 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.351532 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93-plugins-config-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.464669 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.464628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.550784 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.550747 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 02:07:53.556812 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.556776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:53.557205 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:53.557184 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-86995c-5744967cf8-9jc6s"] Sep 14 02:07:55.212388 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:55.212356 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" path="/var/lib/kubelet/pods/b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93/volumes" Sep 14 02:07:55.212903 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:55.212882 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec437c4f-9b5b-43e8-ae08-7352471debde" path="/var/lib/kubelet/pods/ec437c4f-9b5b-43e8-ae08-7352471debde/volumes" Sep 14 02:07:55.839590 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:55.839557 2579 scope.go:117] "RemoveContainer" containerID="3008349681db60f738c5dc2ab58ff2317b0caf40a26948d21840fc1f2e18e0f6" Sep 14 02:07:59.248152 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:59.248106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:59.349655 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:59.349614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:07:59.509688 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:59.509609 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 02:07:59.509874 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:59.509834 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" containerID="cri-o://2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1" gracePeriod=30 Sep 14 02:07:59.531738 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:07:59.531697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:08:04.763134 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.763107 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 02:08:04.856168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856082 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856137 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856168 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856167 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856245 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjhq\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856362 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo\") pod \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\" (UID: \"5019f7c7-e11e-42a5-94a0-9ada88d70fba\") " Sep 14 02:08:04.856443 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data" (OuterVolumeSpecName: "istio-data") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:08:04.856730 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856467 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:08:04.856730 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:08:04.856811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856732 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-data\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.856811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856752 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.856811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856752 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:08:04.856811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856767 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-workload-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.856944 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.856806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:08:04.858482 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.858453 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token" (OuterVolumeSpecName: "istio-token") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:08:04.858482 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.858475 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:08:04.858625 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.858561 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq" (OuterVolumeSpecName: "kube-api-access-xmjhq") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "kube-api-access-xmjhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:08:04.858625 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.858577 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5019f7c7-e11e-42a5-94a0-9ada88d70fba" (UID: "5019f7c7-e11e-42a5-94a0-9ada88d70fba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:08:04.957628 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957597 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istiod-ca-cert\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.957628 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957622 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjhq\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-kube-api-access-xmjhq\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.957628 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957632 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-token\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.957849 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957640 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-credential-socket\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.957849 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957649 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-podinfo\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:04.957849 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:04.957657 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5019f7c7-e11e-42a5-94a0-9ada88d70fba-istio-envoy\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:08:05.286003 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.285967 2579 generic.go:356] "Generic (PLEG): container finished" podID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerID="2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1" exitCode=0 Sep 14 02:08:05.286218 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.286015 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" Sep 14 02:08:05.286218 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.286049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerDied","Data":"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1"} Sep 14 02:08:05.286218 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.286126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs" event={"ID":"5019f7c7-e11e-42a5-94a0-9ada88d70fba","Type":"ContainerDied","Data":"511daaaaa010442450f29657b3bfc8aa1899f6cadcd582219dba44b288d9cdfb"} Sep 14 02:08:05.286218 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.286154 2579 scope.go:117] "RemoveContainer" containerID="2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1" Sep 14 02:08:05.294820 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.294805 2579 scope.go:117] "RemoveContainer" containerID="8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb" Sep 14 02:08:05.302805 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.302783 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 02:08:05.303617 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.303599 2579 scope.go:117] "RemoveContainer" containerID="2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1" Sep 14 02:08:05.303955 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:08:05.303921 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1\": container with ID starting with 2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1 not found: ID does not exist" containerID="2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1" Sep 14 02:08:05.304052 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.303955 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1"} err="failed to get container status \"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1\": rpc error: code = NotFound desc = could not find container \"2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1\": container with ID starting with 2aa39be159560454991dc64d368b34fb2bbfa95daf40a3a7f2bb36c65760ddb1 not found: ID does not exist" Sep 14 02:08:05.304052 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.303981 2579 scope.go:117] "RemoveContainer" containerID="8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb" Sep 14 02:08:05.304713 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:08:05.304629 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb\": container with ID starting with 8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb not found: ID does not exist" containerID="8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb" Sep 14 02:08:05.304713 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.304669 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb"} err="failed to get container status \"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb\": rpc error: code = NotFound desc = could not find container \"8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb\": container with ID starting with 8ea4b056f145dc937f1dcdc4607f86e64976e6315b3b48bfa6e2f10a2ba507eb not found: ID does not exist" Sep 14 02:08:05.306335 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:05.306314 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-86995c-openshift-default-7bbc857fc4-fmnhs"] Sep 14 02:08:07.210525 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:07.210492 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" path="/var/lib/kubelet/pods/5019f7c7-e11e-42a5-94a0-9ada88d70fba/volumes" Sep 14 02:08:17.112045 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:17.111952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:08:50.095811 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:08:50.095767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:09:26.435607 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:09:26.435561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:09:27.878907 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:09:27.878874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:09:27.930366 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:09:27.930338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:09:28.148833 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:09:28.148749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:10:30.710495 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:30.710459 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 02:10:30.713130 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:30.710747 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" podUID="4dc227b5-9c96-4db8-9683-8c3e883deb83" containerName="manager" containerID="cri-o://123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf" gracePeriod=10 Sep 14 02:10:32.874347 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:32.874322 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 02:10:32.968544 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:32.968466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume\") pod \"4dc227b5-9c96-4db8-9683-8c3e883deb83\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " Sep 14 02:10:32.968544 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:32.968524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmhpt\" (UniqueName: \"kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt\") pod \"4dc227b5-9c96-4db8-9683-8c3e883deb83\" (UID: \"4dc227b5-9c96-4db8-9683-8c3e883deb83\") " Sep 14 02:10:32.968826 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:32.968794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4dc227b5-9c96-4db8-9683-8c3e883deb83" (UID: "4dc227b5-9c96-4db8-9683-8c3e883deb83"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:10:32.970442 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:32.970419 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt" (OuterVolumeSpecName: "kube-api-access-pmhpt") pod "4dc227b5-9c96-4db8-9683-8c3e883deb83" (UID: "4dc227b5-9c96-4db8-9683-8c3e883deb83"). InnerVolumeSpecName "kube-api-access-pmhpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:10:33.054430 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.054396 2579 generic.go:356] "Generic (PLEG): container finished" podID="4dc227b5-9c96-4db8-9683-8c3e883deb83" containerID="123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf" exitCode=0 Sep 14 02:10:33.054597 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.054441 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" Sep 14 02:10:33.054597 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.054484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" event={"ID":"4dc227b5-9c96-4db8-9683-8c3e883deb83","Type":"ContainerDied","Data":"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf"} Sep 14 02:10:33.054597 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.054538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k" event={"ID":"4dc227b5-9c96-4db8-9683-8c3e883deb83","Type":"ContainerDied","Data":"93b6b4a329fe869baf4ec646765b2e188e59536c9ea15e074b63812d8e7d418d"} Sep 14 02:10:33.054597 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.054565 2579 scope.go:117] "RemoveContainer" containerID="123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf" Sep 14 02:10:33.063496 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.063479 2579 scope.go:117] "RemoveContainer" containerID="123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf" Sep 14 02:10:33.063785 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:10:33.063762 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf\": container with ID starting with 123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf not found: ID does not exist" containerID="123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf" Sep 14 02:10:33.063836 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.063795 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf"} err="failed to get container status \"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf\": rpc error: code = NotFound desc = could not find container \"123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf\": container with ID starting with 123d5f8ca1c3c876c2da88793ea624795a9d13a844bcdd35e4c829548f6b29cf not found: ID does not exist" Sep 14 02:10:33.069411 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.069391 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dc227b5-9c96-4db8-9683-8c3e883deb83-extensions-socket-volume\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:10:33.069411 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.069410 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhpt\" (UniqueName: \"kubernetes.io/projected/4dc227b5-9c96-4db8-9683-8c3e883deb83-kube-api-access-pmhpt\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:10:33.076032 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.076010 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 02:10:33.080035 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.080013 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mkt6k"] Sep 14 02:10:33.209534 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:33.209496 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc227b5-9c96-4db8-9683-8c3e883deb83" path="/var/lib/kubelet/pods/4dc227b5-9c96-4db8-9683-8c3e883deb83/volumes" Sep 14 02:10:55.395161 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:55.395124 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:10:55.397451 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:10:55.397424 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:11:31.411803 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.411723 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd"] Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412136 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6c78f2-369f-44f9-8f98-f7377ec70405" containerName="revoke-keys" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412149 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6c78f2-369f-44f9-8f98-f7377ec70405" containerName="revoke-keys" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412159 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412164 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412173 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc227b5-9c96-4db8-9683-8c3e883deb83" containerName="manager" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412179 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc227b5-9c96-4db8-9683-8c3e883deb83" containerName="manager" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412185 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412190 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412199 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" containerName="payload-pre-processing" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412204 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" containerName="payload-pre-processing" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412210 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec437c4f-9b5b-43e8-ae08-7352471debde" containerName="payload-processing" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412216 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec437c4f-9b5b-43e8-ae08-7352471debde" containerName="payload-processing" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412274 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ec6501-01d3-4ea2-894e-399f6873534c" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412282 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6c78f2-369f-44f9-8f98-f7377ec70405" containerName="revoke-keys" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412288 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412293 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f0b4d7-d0ab-46e5-af46-dd5d1ba5ce93" containerName="payload-pre-processing" Sep 14 02:11:31.412291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412299 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec437c4f-9b5b-43e8-ae08-7352471debde" containerName="payload-processing" Sep 14 02:11:31.412798 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412307 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc227b5-9c96-4db8-9683-8c3e883deb83" containerName="manager" Sep 14 02:11:31.412798 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.412312 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5019f7c7-e11e-42a5-94a0-9ada88d70fba" containerName="istio-proxy" Sep 14 02:11:31.414942 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.414925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.417628 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.417603 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dccbm\"" Sep 14 02:11:31.429102 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.429080 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd"] Sep 14 02:11:31.482333 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.482300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtj2q\" (UniqueName: \"kubernetes.io/projected/3a27905c-732d-48df-ab5b-91309865b400-kube-api-access-dtj2q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.482333 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.482333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a27905c-732d-48df-ab5b-91309865b400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.583002 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.582964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtj2q\" (UniqueName: \"kubernetes.io/projected/3a27905c-732d-48df-ab5b-91309865b400-kube-api-access-dtj2q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.583002 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.583005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a27905c-732d-48df-ab5b-91309865b400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.583360 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.583343 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a27905c-732d-48df-ab5b-91309865b400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.592938 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.592915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtj2q\" (UniqueName: \"kubernetes.io/projected/3a27905c-732d-48df-ab5b-91309865b400-kube-api-access-dtj2q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xvtpd\" (UID: \"3a27905c-732d-48df-ab5b-91309865b400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:31.725226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:31.725147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:32.002421 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:32.002398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd"] Sep 14 02:11:32.004616 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:11:32.004590 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a27905c_732d_48df_ab5b_91309865b400.slice/crio-9d7867bda9639c917fb46d297441d2d9159ef8e8a71f6a31a8ba5d0f192c6fd1 WatchSource:0}: Error finding container 9d7867bda9639c917fb46d297441d2d9159ef8e8a71f6a31a8ba5d0f192c6fd1: Status 404 returned error can't find the container with id 9d7867bda9639c917fb46d297441d2d9159ef8e8a71f6a31a8ba5d0f192c6fd1 Sep 14 02:11:32.353255 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:32.353171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" event={"ID":"3a27905c-732d-48df-ab5b-91309865b400","Type":"ContainerStarted","Data":"77edbb9263d386eb956e5832201dae5ca28ca0d33e1ad63cc9a37fa1e2138077"} Sep 14 02:11:32.353255 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:32.353223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" event={"ID":"3a27905c-732d-48df-ab5b-91309865b400","Type":"ContainerStarted","Data":"9d7867bda9639c917fb46d297441d2d9159ef8e8a71f6a31a8ba5d0f192c6fd1"} Sep 14 02:11:32.353442 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:32.353282 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:32.376744 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:32.376690 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" podStartSLOduration=1.3766700840000001 podStartE2EDuration="1.376670084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:11:32.374525833 +0000 UTC m=+2437.607891091" watchObservedRunningTime="2026-09-14 02:11:32.376670084 +0000 UTC m=+2437.610035342" Sep 14 02:11:43.360661 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:43.360628 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xvtpd" Sep 14 02:11:55.923725 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:11:55.923692 2579 scope.go:117] "RemoveContainer" containerID="117188a797ec571fdf9fc35b81adfbb0978185ea3faba38b2e8a837a75fbcf5b" Sep 14 02:12:23.425164 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.425127 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:12:23.429363 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.429344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.432205 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.432180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-106ff464-kserve-self-signed-certs\"" Sep 14 02:12:23.441020 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.440998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:12:23.494784 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.494750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:12:23.555266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rr97\" (UniqueName: \"kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.555266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.555536 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.555536 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.555536 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.555536 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.555463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656646 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rr97\" (UniqueName: \"kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656646 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656903 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656903 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656903 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.656903 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.656747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.657161 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.657123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.657161 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.657138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.657261 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.657233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.659027 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.659005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.659210 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.659190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.666472 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.666439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rr97\" (UniqueName: \"kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97\") pod \"e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:23.740433 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:23.740403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:24.009489 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:24.009416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:12:24.013273 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:12:24.013243 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6815c47_a247_4790_89ce_a2d34dd15ffa.slice/crio-fe390ef466ae83744cc4b96cba91319cfb11b028a00bf4a131773c303fc25b6e WatchSource:0}: Error finding container fe390ef466ae83744cc4b96cba91319cfb11b028a00bf4a131773c303fc25b6e: Status 404 returned error can't find the container with id fe390ef466ae83744cc4b96cba91319cfb11b028a00bf4a131773c303fc25b6e Sep 14 02:12:24.612854 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:24.612820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" event={"ID":"e6815c47-a247-4790-89ce-a2d34dd15ffa","Type":"ContainerStarted","Data":"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7"} Sep 14 02:12:24.613264 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:24.612861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" event={"ID":"e6815c47-a247-4790-89ce-a2d34dd15ffa","Type":"ContainerStarted","Data":"fe390ef466ae83744cc4b96cba91319cfb11b028a00bf4a131773c303fc25b6e"} Sep 14 02:12:24.631393 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:24.631346 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" podStartSLOduration=1.6313319480000001 podStartE2EDuration="1.631331948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:12:24.630429915 +0000 UTC m=+2489.863795195" watchObservedRunningTime="2026-09-14 02:12:24.631331948 +0000 UTC m=+2489.864697205" Sep 14 02:12:27.000935 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:27.000889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:12:32.845414 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:32.845381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:12:33.740709 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:33.740677 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:33.740939 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:33.740723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:33.753564 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:33.753534 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:34.682384 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:34.682356 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:12:50.309425 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:50.309323 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:12:50.326837 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:50.326813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:12:51.136453 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:51.136413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:12:55.944874 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:12:55.944839 2579 scope.go:117] "RemoveContainer" containerID="2783f8cf86df5e14066923bd2e5adf1dba95534c5e6770cf7e28ff305d724cfb" Sep 14 02:13:05.329523 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.329461 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" podUID="e6815c47-a247-4790-89ce-a2d34dd15ffa" containerName="main" containerID="cri-o://96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7" gracePeriod=45 Sep 14 02:13:05.576704 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.576680 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:13:05.659432 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659340 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rr97\" (UniqueName: \"kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659432 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659432 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659720 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659720 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659720 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir\") pod \"e6815c47-a247-4790-89ce-a2d34dd15ffa\" (UID: \"e6815c47-a247-4790-89ce-a2d34dd15ffa\") " Sep 14 02:13:05.659720 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659673 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:13:05.659720 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659701 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home" (OuterVolumeSpecName: "home") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:13:05.659967 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659845 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:13:05.659967 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659854 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-model-cache\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.659967 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.659902 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-home\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.661786 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.661758 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:13:05.661786 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.661773 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm" (OuterVolumeSpecName: "dshm") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:13:05.661786 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.661774 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97" (OuterVolumeSpecName: "kube-api-access-2rr97") pod "e6815c47-a247-4790-89ce-a2d34dd15ffa" (UID: "e6815c47-a247-4790-89ce-a2d34dd15ffa"). InnerVolumeSpecName "kube-api-access-2rr97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:13:05.761048 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.761004 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rr97\" (UniqueName: \"kubernetes.io/projected/e6815c47-a247-4790-89ce-a2d34dd15ffa-kube-api-access-2rr97\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.761048 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.761043 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6815c47-a247-4790-89ce-a2d34dd15ffa-tls-certs\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.761048 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.761053 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-dshm\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.761048 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.761087 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6815c47-a247-4790-89ce-a2d34dd15ffa-tmp-dir\") on node \"ip-10-0-130-123.ec2.internal\" DevicePath \"\"" Sep 14 02:13:05.840860 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.840819 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6815c47-a247-4790-89ce-a2d34dd15ffa" containerID="96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7" exitCode=0 Sep 14 02:13:05.841038 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.840865 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" Sep 14 02:13:05.841038 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.840876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" event={"ID":"e6815c47-a247-4790-89ce-a2d34dd15ffa","Type":"ContainerDied","Data":"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7"} Sep 14 02:13:05.841038 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.840909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz" event={"ID":"e6815c47-a247-4790-89ce-a2d34dd15ffa","Type":"ContainerDied","Data":"fe390ef466ae83744cc4b96cba91319cfb11b028a00bf4a131773c303fc25b6e"} Sep 14 02:13:05.841038 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.840924 2579 scope.go:117] "RemoveContainer" containerID="96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7" Sep 14 02:13:05.850485 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.850462 2579 scope.go:117] "RemoveContainer" containerID="96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7" Sep 14 02:13:05.850759 ip-10-0-130-123 kubenswrapper[2579]: E0914 02:13:05.850726 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7\": container with ID starting with 96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7 not found: ID does not exist" containerID="96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7" Sep 14 02:13:05.850829 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.850765 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7"} err="failed to get container status \"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7\": rpc error: code = NotFound desc = could not find container \"96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7\": container with ID starting with 96b42a4886d32d335611bcac74d29d7cdaac71ecd7e959ca6deff1741870bca7 not found: ID does not exist" Sep 14 02:13:05.868476 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.868447 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:13:05.870046 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:05.870027 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-106ff464-kserve-7b8cbcdd59-4w5dz"] Sep 14 02:13:07.210571 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:07.210532 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6815c47-a247-4790-89ce-a2d34dd15ffa" path="/var/lib/kubelet/pods/e6815c47-a247-4790-89ce-a2d34dd15ffa/volumes" Sep 14 02:13:22.208964 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:22.208923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:22.260411 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:22.260373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:31.208804 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:31.208755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:44.311967 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:44.311937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5fngv_3328ae26-849a-41b2-9b10-a7553c33225e/server/0.log" Sep 14 02:13:44.424761 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:44.424736 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zcgrr_c63a1f82-cca1-4079-b45b-b108955c1fb3/manager/0.log" Sep 14 02:13:44.758926 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:44.758897 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-mxcgl_ec030c39-7709-474e-b00d-894aede051f0/copy-manifests/0.log" Sep 14 02:13:44.765291 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:44.765269 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-mxcgl_ec030c39-7709-474e-b00d-894aede051f0/manager/0.log" Sep 14 02:13:45.305185 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:45.305152 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29822505-gnq2j_5427eb6b-c262-4452-ae79-9c3be2919231/cleanup/0.log" Sep 14 02:13:45.413012 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:45.412982 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29822520-zj898_de065a5e-f36b-474b-b9ad-138690e35466/cleanup/0.log" Sep 14 02:13:46.349739 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:46.349711 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sbs6s_f6e1e456-1af0-4374-b430-43b2088e9486/manager/0.log" Sep 14 02:13:46.668918 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:46.668844 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xvtpd_3a27905c-732d-48df-ab5b-91309865b400/manager/0.log" Sep 14 02:13:47.243670 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:47.243641 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n_abf63e43-a015-4240-a64b-66885f55222d/istio-proxy/0.log" Sep 14 02:13:47.353664 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:47.353637 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hx8sl_aeb8325c-5c39-46bc-bab7-94190f37ef43/discovery/0.log" Sep 14 02:13:47.453429 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:47.453401 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb7b7df59-cm6xg_9426f359-66ae-482d-8573-bad9ffb4cb48/kube-auth-proxy/0.log" Sep 14 02:13:47.780553 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:47.780520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9twnd_62e8bd73-3af1-471c-b362-427bfdcbf437/payload-pre-processing/0.log" Sep 14 02:13:48.347862 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:48.347837 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rv599_e97394f1-ef43-4919-8a9d-118cb991dddb/main/0.log" Sep 14 02:13:48.450966 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:48.450939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xpnn5_66c6d581-1156-4101-9860-32ebfe300c5c/main/0.log" Sep 14 02:13:51.193711 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:51.193678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:51.294915 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:51.294884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:54.448569 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.448531 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7nzg8/must-gather-sz8qm"] Sep 14 02:13:54.448965 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.448933 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6815c47-a247-4790-89ce-a2d34dd15ffa" containerName="main" Sep 14 02:13:54.448965 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.448945 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6815c47-a247-4790-89ce-a2d34dd15ffa" containerName="main" Sep 14 02:13:54.449048 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.449016 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6815c47-a247-4790-89ce-a2d34dd15ffa" containerName="main" Sep 14 02:13:54.452366 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.452345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.454957 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.454934 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7nzg8\"/\"kube-root-ca.crt\"" Sep 14 02:13:54.455776 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.455758 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7nzg8\"/\"default-dockercfg-nvqmh\"" Sep 14 02:13:54.455776 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.455772 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7nzg8\"/\"openshift-service-ca.crt\"" Sep 14 02:13:54.459747 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.459724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:54.460642 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.460619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/must-gather-sz8qm"] Sep 14 02:13:54.556280 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.556247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" limit=1024 Sep 14 02:13:54.598568 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.598537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwvk\" (UniqueName: \"kubernetes.io/projected/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-kube-api-access-sfwvk\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.598696 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.598584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-must-gather-output\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.700124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.700034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwvk\" (UniqueName: \"kubernetes.io/projected/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-kube-api-access-sfwvk\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.700124 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.700101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-must-gather-output\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.700402 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.700386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-must-gather-output\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.718013 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.717988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwvk\" (UniqueName: \"kubernetes.io/projected/e624b79a-ff6c-453f-a67a-5ba4ef14a5c1-kube-api-access-sfwvk\") pod \"must-gather-sz8qm\" (UID: \"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1\") " pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:54.762770 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:54.762745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" Sep 14 02:13:55.032872 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:55.032814 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/must-gather-sz8qm"] Sep 14 02:13:55.035045 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:13:55.035018 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode624b79a_ff6c_453f_a67a_5ba4ef14a5c1.slice/crio-fca94af13bda1f08d9251f0154a92899fc8a089cee0e16e7937f27894e524c57 WatchSource:0}: Error finding container fca94af13bda1f08d9251f0154a92899fc8a089cee0e16e7937f27894e524c57: Status 404 returned error can't find the container with id fca94af13bda1f08d9251f0154a92899fc8a089cee0e16e7937f27894e524c57 Sep 14 02:13:55.036932 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:55.036916 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 02:13:55.092130 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:55.092103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" event={"ID":"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1","Type":"ContainerStarted","Data":"fca94af13bda1f08d9251f0154a92899fc8a089cee0e16e7937f27894e524c57"} Sep 14 02:13:55.964633 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:55.964601 2579 scope.go:117] "RemoveContainer" containerID="0589a203bac23a1f3f6b8bbc3ed1c747542bb177f968e65d56906316965a98bd" Sep 14 02:13:56.005221 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:56.005198 2579 scope.go:117] "RemoveContainer" containerID="6b12223a8803e5233449e27d1be8391eb87f40c360547e3f693268868a50a3f3" Sep 14 02:13:56.012927 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:56.012904 2579 scope.go:117] "RemoveContainer" containerID="a93db4b528c39f746bfa1718c53451d2b8fd12a0532f628cf2b35e4f0afcfd10" Sep 14 02:13:57.109409 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.109361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" event={"ID":"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1","Type":"ContainerStarted","Data":"f5469b863a5ef9275a3e1eca6c61ea03eccd193884efe0862339e37d0c4c9d55"} Sep 14 02:13:57.109894 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.109723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" event={"ID":"e624b79a-ff6c-453f-a67a-5ba4ef14a5c1","Type":"ContainerStarted","Data":"4a80bf22005a2fd35cc7075f38efa27dbf0edadb1c9571de6f583777d2f692ad"} Sep 14 02:13:57.127878 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.127820 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7nzg8/must-gather-sz8qm" podStartSLOduration=2.104156273 podStartE2EDuration="3.127800766s" totalImagesPullingTime="1.023644493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:13:57.12442915 +0000 UTC m=+2582.357794409" watchObservedRunningTime="2026-09-14 02:13:57.127800766 +0000 UTC m=+2582.361166024" Sep 14 02:13:57.632466 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.632436 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q8gjc_c3e106f2-e5ce-4769-b070-073ce8dee7ac/global-pull-secret-syncer/0.log" Sep 14 02:13:57.683599 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.683569 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hr82d_aac06f78-7b86-47c6-a226-4a77d6e7ba68/konnectivity-agent/0.log" Sep 14 02:13:57.762308 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:13:57.762279 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-123.ec2.internal_57ac709050433b22e72b53986d3d835e/haproxy/0.log" Sep 14 02:14:02.624426 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:02.624371 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sbs6s_f6e1e456-1af0-4374-b430-43b2088e9486/manager/0.log" Sep 14 02:14:02.721710 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:02.721676 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-xvtpd_3a27905c-732d-48df-ab5b-91309865b400/manager/0.log" Sep 14 02:14:04.344172 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.344137 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/alertmanager/0.log" Sep 14 02:14:04.369616 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.369584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/config-reloader/0.log" Sep 14 02:14:04.390840 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.390808 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/kube-rbac-proxy-web/0.log" Sep 14 02:14:04.412796 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.412764 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/kube-rbac-proxy/0.log" Sep 14 02:14:04.436255 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.436223 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/kube-rbac-proxy-metric/0.log" Sep 14 02:14:04.458746 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.458715 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/prom-label-proxy/0.log" Sep 14 02:14:04.483382 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.483349 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b546adde-3637-4d5b-a529-902fe62ebed0/init-config-reloader/0.log" Sep 14 02:14:04.580414 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.580307 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cm5pw_b4c12dc5-82b4-40e7-a017-2aceb2e305ab/kube-state-metrics/0.log" Sep 14 02:14:04.605888 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.605796 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cm5pw_b4c12dc5-82b4-40e7-a017-2aceb2e305ab/kube-rbac-proxy-main/0.log" Sep 14 02:14:04.630232 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.630188 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-cm5pw_b4c12dc5-82b4-40e7-a017-2aceb2e305ab/kube-rbac-proxy-self/0.log" Sep 14 02:14:04.894455 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.894365 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2v2m_e7dd27df-a1ba-40c0-ac7e-2844aca312bc/node-exporter/0.log" Sep 14 02:14:04.913851 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.913820 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2v2m_e7dd27df-a1ba-40c0-ac7e-2844aca312bc/kube-rbac-proxy/0.log" Sep 14 02:14:04.938202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:04.938172 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2v2m_e7dd27df-a1ba-40c0-ac7e-2844aca312bc/init-textfile/0.log" Sep 14 02:14:05.275832 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:05.275791 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-rmnzl_20670962-98ca-454a-a115-638fce2ade07/prometheus-operator-admission-webhook/0.log" Sep 14 02:14:07.479234 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.479202 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/1.log" Sep 14 02:14:07.485287 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.485260 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-9c9j2_fd2039fe-5e89-469b-9995-a28af90bc447/console-operator/2.log" Sep 14 02:14:07.891895 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.891798 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74"] Sep 14 02:14:07.899366 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.899335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:07.903018 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.902983 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74"] Sep 14 02:14:07.943550 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.943511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlczc\" (UniqueName: \"kubernetes.io/projected/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-kube-api-access-jlczc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:07.943816 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.943795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-proc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:07.943981 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.943967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-podres\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:07.944166 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.944150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-lib-modules\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:07.944288 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:07.944275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-sys\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.045923 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.045868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-podres\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046156 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.045977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-lib-modules\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046156 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-sys\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046156 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-podres\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlczc\" (UniqueName: \"kubernetes.io/projected/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-kube-api-access-jlczc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-lib-modules\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-sys\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046345 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-proc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.046568 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.046357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-proc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.055121 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.055094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlczc\" (UniqueName: \"kubernetes.io/projected/3b5c5f04-df3c-461e-89f0-ef1f7cfab582-kube-api-access-jlczc\") pod \"perf-node-gather-daemonset-nwl74\" (UID: \"3b5c5f04-df3c-461e-89f0-ef1f7cfab582\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.215966 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.215876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:08.500897 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.500860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74"] Sep 14 02:14:08.502215 ip-10-0-130-123 kubenswrapper[2579]: W0914 02:14:08.502183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b5c5f04_df3c_461e_89f0_ef1f7cfab582.slice/crio-d8ed2c7225f5a4469d199897b0c7a0bb2e28a958ab43ed46606bf2cbdeea5f67 WatchSource:0}: Error finding container d8ed2c7225f5a4469d199897b0c7a0bb2e28a958ab43ed46606bf2cbdeea5f67: Status 404 returned error can't find the container with id d8ed2c7225f5a4469d199897b0c7a0bb2e28a958ab43ed46606bf2cbdeea5f67 Sep 14 02:14:08.774912 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:08.774839 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-n4f99_a5392461-ac66-46c3-96b3-dc17eb4dac87/volume-data-source-validator/0.log" Sep 14 02:14:09.199406 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.199322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" event={"ID":"3b5c5f04-df3c-461e-89f0-ef1f7cfab582","Type":"ContainerStarted","Data":"77ad29b2872881c2050b9b76f9289d2afc06b6333d2da03448e6f4050e0eeb78"} Sep 14 02:14:09.199406 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.199398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" event={"ID":"3b5c5f04-df3c-461e-89f0-ef1f7cfab582","Type":"ContainerStarted","Data":"d8ed2c7225f5a4469d199897b0c7a0bb2e28a958ab43ed46606bf2cbdeea5f67"} Sep 14 02:14:09.200226 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.200196 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:09.215221 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.215172 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" podStartSLOduration=2.21515309 podStartE2EDuration="2.21515309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:14:09.214589337 +0000 UTC m=+2594.447954614" watchObservedRunningTime="2026-09-14 02:14:09.21515309 +0000 UTC m=+2594.448518348" Sep 14 02:14:09.793262 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.793216 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vzjs_685a7a90-19bb-49dc-b5df-54ae3937a55a/dns/0.log" Sep 14 02:14:09.813078 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.813031 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vzjs_685a7a90-19bb-49dc-b5df-54ae3937a55a/kube-rbac-proxy/0.log" Sep 14 02:14:09.967183 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:09.967158 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szkpn_1006b6cf-ffe6-49bb-afdd-c86e5dd32791/dns-node-resolver/0.log" Sep 14 02:14:10.495700 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:10.495649 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bb7f9cb9c-fzspk_e3d93b1f-85ea-4e71-8220-c90b277f74d5/registry/0.log" Sep 14 02:14:10.515642 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:10.515619 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jvncq_5eac84e6-0163-4182-a8ab-63e166b09d6f/node-ca/0.log" Sep 14 02:14:11.515210 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:11.515164 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n_abf63e43-a015-4240-a64b-66885f55222d/istio-proxy/0.log" Sep 14 02:14:12.055634 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:12.055595 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hx8sl_aeb8325c-5c39-46bc-bab7-94190f37ef43/discovery/0.log" Sep 14 02:14:12.080667 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:12.080641 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb7b7df59-cm6xg_9426f359-66ae-482d-8573-bad9ffb4cb48/kube-auth-proxy/0.log" Sep 14 02:14:12.166543 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:12.166515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9twnd_62e8bd73-3af1-471c-b362-427bfdcbf437/payload-pre-processing/0.log" Sep 14 02:14:13.085808 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:13.085782 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n646v_6153521d-a4f5-4e75-9ce8-c477938c92d5/serve-healthcheck-canary/0.log" Sep 14 02:14:13.637542 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:13.637519 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64vfb_4006e721-807a-4462-a3a0-ec6da9413fd4/kube-rbac-proxy/0.log" Sep 14 02:14:13.659769 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:13.659740 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64vfb_4006e721-807a-4462-a3a0-ec6da9413fd4/exporter/0.log" Sep 14 02:14:13.683586 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:13.683552 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-64vfb_4006e721-807a-4462-a3a0-ec6da9413fd4/extractor/0.log" Sep 14 02:14:16.086895 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:16.086869 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5fngv_3328ae26-849a-41b2-9b10-a7553c33225e/server/0.log" Sep 14 02:14:16.196293 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:16.196256 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zcgrr_c63a1f82-cca1-4079-b45b-b108955c1fb3/manager/0.log" Sep 14 02:14:16.223815 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:16.223784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-nwl74" Sep 14 02:14:16.361744 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:16.361663 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-mxcgl_ec030c39-7709-474e-b00d-894aede051f0/manager/0.log" Sep 14 02:14:16.381210 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:16.381184 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-mxcgl_ec030c39-7709-474e-b00d-894aede051f0/copy-manifests/0.log" Sep 14 02:14:17.603400 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:17.603369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fzfct_63fc0a50-cfb5-40ad-b513-8502721f2632/openshift-lws-operator/0.log" Sep 14 02:14:17.797073 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:17.797012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n_abf63e43-a015-4240-a64b-66885f55222d/istio-proxy/0.log" Sep 14 02:14:18.426884 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:18.426479 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hx8sl_aeb8325c-5c39-46bc-bab7-94190f37ef43/discovery/0.log" Sep 14 02:14:18.437348 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:18.437313 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb7b7df59-cm6xg_9426f359-66ae-482d-8573-bad9ffb4cb48/kube-auth-proxy/0.log" Sep 14 02:14:18.505080 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:18.505017 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9twnd_62e8bd73-3af1-471c-b362-427bfdcbf437/payload-pre-processing/0.log" Sep 14 02:14:20.217373 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:20.217346 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zpl4n_abf63e43-a015-4240-a64b-66885f55222d/istio-proxy/0.log" Sep 14 02:14:20.834202 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:20.834172 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hx8sl_aeb8325c-5c39-46bc-bab7-94190f37ef43/discovery/0.log" Sep 14 02:14:20.854968 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:20.854934 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb7b7df59-cm6xg_9426f359-66ae-482d-8573-bad9ffb4cb48/kube-auth-proxy/0.log" Sep 14 02:14:20.919848 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:20.919815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9twnd_62e8bd73-3af1-471c-b362-427bfdcbf437/payload-pre-processing/0.log" Sep 14 02:14:23.043539 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:23.043505 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-k55dd_12c4c6ea-daca-412b-afd5-981960e122e5/migrator/0.log" Sep 14 02:14:23.065091 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:23.065051 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-k55dd_12c4c6ea-daca-412b-afd5-981960e122e5/graceful-termination/0.log" Sep 14 02:14:23.408715 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:23.408621 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-tqpjh_1edd9088-2b01-469a-abd3-e05031c14dee/kube-storage-version-migrator-operator/1.log" Sep 14 02:14:23.409850 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:23.409825 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-tqpjh_1edd9088-2b01-469a-abd3-e05031c14dee/kube-storage-version-migrator-operator/0.log" Sep 14 02:14:24.422725 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.422699 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/kube-multus-additional-cni-plugins/0.log" Sep 14 02:14:24.442623 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.442596 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/egress-router-binary-copy/0.log" Sep 14 02:14:24.463032 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.463010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/cni-plugins/0.log" Sep 14 02:14:24.482350 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.482329 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/bond-cni-plugin/0.log" Sep 14 02:14:24.500816 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.500795 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/routeoverride-cni/0.log" Sep 14 02:14:24.520165 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.520118 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/whereabouts-cni-bincopy/0.log" Sep 14 02:14:24.539632 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.539614 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5qs9v_50a0826b-6cf4-4e18-8a26-6d5e08b1d953/whereabouts-cni/0.log" Sep 14 02:14:24.939277 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:24.939241 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pzh7k_1e9dc715-928b-4cef-ba0d-8cb89739cf45/kube-multus/0.log" Sep 14 02:14:25.045827 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:25.045801 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xhtx9_d5450a90-678e-4192-ab47-81dcef253c6c/network-metrics-daemon/0.log" Sep 14 02:14:25.061420 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:25.061397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xhtx9_d5450a90-678e-4192-ab47-81dcef253c6c/kube-rbac-proxy/0.log" Sep 14 02:14:26.283913 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.283879 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/ovn-controller/0.log" Sep 14 02:14:26.300027 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.299996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/ovn-acl-logging/0.log" Sep 14 02:14:26.326266 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.326234 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/kube-rbac-proxy-node/0.log" Sep 14 02:14:26.348780 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.348750 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/kube-rbac-proxy-ovn-metrics/0.log" Sep 14 02:14:26.365655 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.365630 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/northd/0.log" Sep 14 02:14:26.386096 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.386051 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/nbdb/0.log" Sep 14 02:14:26.406665 ip-10-0-130-123 kubenswrapper[2579]: I0914 02:14:26.406638 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wzrrl_87b63fc3-968d-40ea-878e-dc6a1801db68/sbdb/0.log"