Jul 15 19:56:32.177337 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 19:56:32.177499 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 19:56:32.177516 ip-10-0-141-87 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 19:56:32.177947 ip-10-0-141-87 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 19:56:42.357600 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 19:56:42.357622 ip-10-0-141-87 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1771e1690af444d9aafb415f3998774b -- Jul 15 19:58:41.606172 ip-10-0-141-87 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:58:42.009960 ip-10-0-141-87 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:58:42.009960 ip-10-0-141-87 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:58:42.009960 ip-10-0-141-87 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:58:42.009960 ip-10-0-141-87 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:58:42.009960 ip-10-0-141-87 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:58:42.012653 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.012459 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:58:42.020539 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020509 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.020539 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020533 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.020539 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020537 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.020539 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020540 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.020539 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020543 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020546 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020549 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020552 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020561 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020567 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020569 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020572 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020575 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020578 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020581 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020584 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020586 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020590 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020593 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020595 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020598 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020601 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020604 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020607 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.020699 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020610 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020612 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020615 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020617 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020620 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020623 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020626 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020628 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020633 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020636 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020642 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020645 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020649 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020652 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020655 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020659 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020662 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020664 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020667 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020669 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.021196 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020672 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020675 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020678 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020681 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020683 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020686 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020688 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020691 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020693 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020696 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020699 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020702 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020704 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020707 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020710 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020714 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020716 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020719 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020721 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020724 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.021734 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020727 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020730 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020732 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020735 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020737 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020740 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020744 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020746 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020750 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020752 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020754 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020757 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020759 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020762 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020766 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020770 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020774 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020777 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020780 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.022235 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020782 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020785 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020788 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020790 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020793 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020795 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020798 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020800 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020803 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020805 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020809 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020812 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020814 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020817 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020821 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.020837 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021600 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021608 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021611 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021614 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.022701 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021616 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021619 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021623 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021626 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021629 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021631 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021634 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021636 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021639 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021643 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021647 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021650 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021652 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021655 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021658 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021660 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021663 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021666 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021668 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021671 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.023449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021674 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021676 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021679 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021682 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021685 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021688 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021690 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021693 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021696 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021698 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021701 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021703 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021706 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021708 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021711 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021715 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021718 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021720 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021723 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021725 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.024212 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021728 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021732 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021736 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021739 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021742 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021744 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021748 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021750 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021753 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021756 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021758 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021761 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021763 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021766 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021769 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021771 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021774 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021777 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021780 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.024709 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021783 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021786 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021789 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021791 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021794 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021796 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021799 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021801 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021804 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021806 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021809 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021811 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021814 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021816 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021819 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021839 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021844 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021847 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021851 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.025190 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021853 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021857 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021860 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021863 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021866 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021870 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021873 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021875 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021878 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021882 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021885 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021887 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021891 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021894 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021897 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021899 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.021902 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022312 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022317 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022320 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.025661 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022323 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022326 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022330 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022332 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022335 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022338 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022341 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022344 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022346 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022349 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022352 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022354 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022357 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022359 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022362 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022364 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022367 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022369 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022372 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022374 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.026183 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022378 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022380 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022383 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022385 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022388 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022391 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022394 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022396 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022399 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022402 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022404 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022407 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022409 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022412 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022414 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022417 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022420 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022423 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022425 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022428 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.026760 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022431 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022433 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022436 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022438 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022441 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022444 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022448 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022452 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022455 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022458 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022460 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022463 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022465 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022468 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022470 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022473 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022476 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022480 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022484 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.027329 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022487 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022492 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022495 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022498 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022502 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022504 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022507 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022509 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022512 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022515 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022517 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022520 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022522 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022526 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022528 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022531 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022533 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022536 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022538 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022541 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.027797 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022544 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022547 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022551 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022554 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022557 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022559 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022562 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022565 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022567 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022570 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022573 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022575 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022578 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022582 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022585 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022587 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.022590 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.022687 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.022708 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024195 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:58:42.028308 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024204 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024216 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024238 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024249 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024255 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024259 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024266 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024270 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024274 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024277 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024281 2576 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024284 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024287 2576 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024291 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024295 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024303 2576 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024306 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024310 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024313 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024317 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024322 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024326 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024329 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:58:42.028863 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024336 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024339 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024342 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024345 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024349 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024352 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024358 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024361 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024365 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024371 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024375 2576 flags.go:64] FLAG: --enable-server="true" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024378 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024385 2576 flags.go:64] FLAG: --event-burst="100" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024388 2576 flags.go:64] FLAG: --event-qps="50" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024392 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024395 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024399 2576 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024405 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024409 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024412 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024415 2576 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024419 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024422 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024425 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024428 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:58:42.029409 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024432 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024437 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024441 2576 flags.go:64] FLAG: --feature-gates="" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024444 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024448 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024451 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024455 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024458 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024461 2576 flags.go:64] FLAG: --help="false" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024467 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024471 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024474 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024477 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024480 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024486 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024490 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024493 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024496 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024502 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024505 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024509 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024513 2576 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024516 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024519 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:58:42.030055 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024523 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024525 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024531 2576 flags.go:64] FLAG: --lock-file="" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024534 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024537 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024541 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024547 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024551 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024554 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024557 2576 flags.go:64] FLAG: --logging-format="text" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024560 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024566 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024569 2576 flags.go:64] FLAG: --manifest-url="" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024572 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024577 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024580 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024585 2576 flags.go:64] FLAG: --max-pods="110" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024588 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024591 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024597 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024600 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024604 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024608 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024614 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024623 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:58:42.030637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024627 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024633 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024637 2576 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024640 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024647 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024650 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024653 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024657 2576 flags.go:64] FLAG: --port="10250" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024660 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024664 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09b525ef188ea85a9" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024670 2576 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024673 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024676 2576 flags.go:64] FLAG: --register-node="true" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024680 2576 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024684 2576 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024687 2576 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024690 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024693 2576 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024699 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024703 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024706 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024710 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024713 2576 flags.go:64] FLAG: --runonce="false" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024716 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024720 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:58:42.031263 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024724 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024729 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024732 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024735 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024738 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024742 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024745 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024749 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024752 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024755 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024760 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024764 2576 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024767 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024774 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024777 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024780 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024784 2576 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024790 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024793 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024796 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024799 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024802 2576 flags.go:64] FLAG: --v="2" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024807 2576 flags.go:64] FLAG: --version="false" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024811 2576 flags.go:64] FLAG: --vmodule="" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024815 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:58:42.031941 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.024821 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025066 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025074 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025077 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025081 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025084 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025086 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025089 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025092 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025095 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025099 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025101 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025104 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025107 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025110 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025116 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025119 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025121 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025124 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025127 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.032602 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025131 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025134 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025136 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025139 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025142 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025144 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025147 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025152 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025155 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025158 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025161 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025163 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025166 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025169 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025172 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025174 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025177 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025180 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025183 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025185 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.033109 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025190 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025193 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025196 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025199 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025202 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025205 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025208 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025211 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025214 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025217 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025219 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025222 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025226 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025231 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025236 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025240 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025243 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025247 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025249 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.033592 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025252 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025256 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025260 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025264 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025268 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025274 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025277 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025282 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025285 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025288 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025291 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025294 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025297 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025300 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025302 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025305 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025308 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025311 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025316 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.034061 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025319 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025322 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025324 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025327 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025330 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025333 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025335 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025338 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025341 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025344 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025347 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025353 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025355 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025358 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025361 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025364 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025366 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025369 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025372 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025374 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.034521 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025379 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.025382 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.026057 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.033692 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.033711 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.033733 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.033739 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033800 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033806 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033809 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033813 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033817 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033822 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033841 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.035027 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033844 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033847 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033850 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033852 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033855 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033858 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033861 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033864 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033868 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033871 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033874 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033877 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033880 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033883 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033885 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033888 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033891 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033894 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033898 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.035388 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033902 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033904 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033907 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033911 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033914 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033917 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033920 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033923 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033925 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033928 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033931 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033934 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033936 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033939 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033942 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033944 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033947 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033951 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033954 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033956 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.035953 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033959 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033961 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033964 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033966 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033969 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033972 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033975 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033979 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033982 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033985 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033987 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033990 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033992 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033995 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.033998 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034002 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034004 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034007 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034010 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034013 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.036449 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034015 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034018 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034020 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034023 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034025 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034028 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034030 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034033 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034036 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034038 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034041 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034043 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034046 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034048 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034051 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034054 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034056 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034059 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034061 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034065 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.036965 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034067 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034070 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034073 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034076 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034078 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034081 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034083 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034087 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034090 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034093 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034096 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034099 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034101 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.034107 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034229 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:58:42.037454 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034234 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034238 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034241 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034244 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034246 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034249 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034251 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034254 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034256 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034261 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034265 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034268 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034271 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034273 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034276 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034280 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034282 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034285 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034288 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:58:42.037843 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034290 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034293 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034296 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034298 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034301 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034304 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034308 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034311 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034313 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034316 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034318 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034321 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034323 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034326 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034330 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034334 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034337 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034340 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034343 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:58:42.038297 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034346 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034348 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034351 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034353 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034356 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034358 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034361 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034363 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034366 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034368 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034371 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034374 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034377 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034379 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034382 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034384 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034387 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034389 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034393 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034396 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:58:42.038767 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034400 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034402 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034405 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034407 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034410 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034413 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034415 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034418 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034420 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034423 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034425 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034427 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034430 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034432 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034435 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034437 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034440 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034444 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034448 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034450 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:58:42.039262 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034453 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034455 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034458 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034461 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034464 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034466 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034469 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034471 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034474 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034476 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034480 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034485 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034488 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034492 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034495 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034497 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034500 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034503 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034505 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:58:42.039747 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:42.034508 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.034513 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.034662 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.037209 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.038684 2576 server.go:1019] "Starting client certificate rotation" Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.038793 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:58:42.040307 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.038852 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:58:42.041681 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.041663 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:58:42.042967 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.042933 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:58:42.051219 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.051194 2576 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:58:42.055404 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.055381 2576 log.go:25] "Validated CRI v1 image API" Jul 15 19:58:42.057095 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.057072 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:58:42.059479 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.059454 2576 fs.go:135] Filesystem UUIDs: map[4fd7c4dc-ad0c-42ab-9871-b1c224529fa2:/dev/nvme0n1p4 5f7ee5d5-6221-4896-9b68-7e2b736195f4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 15 19:58:42.059579 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.059477 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:58:42.065925 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.065796 2576 manager.go:216] Machine: {Timestamp:2026-07-15 19:58:42.064570593 +0000 UTC m=+0.335320085 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3069048 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2392ec58ac1d4be9d1694d192ea092 SystemUUID:ec2392ec-58ac-1d4b-e9d1-694d192ea092 BootID:1771e169-0af4-44d9-aafb-415f3998774b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:7a:d1:02:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:7a:d1:02:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:be:72:96:90:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:58:42.065925 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.065917 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:58:42.066041 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.066016 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:58:42.067753 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.067711 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:58:42.067920 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.067756 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:58:42.068864 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.068851 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:58:42.068900 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.068868 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:58:42.068900 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.068876 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:58:42.069525 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.069513 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:58:42.069699 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.069685 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:58:42.070874 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.070861 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:58:42.073703 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073689 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:58:42.073861 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073850 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:58:42.073907 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073887 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:58:42.073907 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073899 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:58:42.073963 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073911 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:58:42.073963 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073921 2576 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:58:42.073963 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.073931 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:58:42.074324 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.074311 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:58:42.074376 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.074330 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:58:42.076580 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.076558 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:58:42.078625 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078610 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:58:42.078682 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078643 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:58:42.078847 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078836 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:58:42.078888 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078853 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:58:42.078888 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078863 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:58:42.078888 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078872 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:58:42.078888 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078885 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:58:42.078992 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.078891 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:58:42.079310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079301 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:58:42.079310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079310 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:58:42.079371 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079318 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:58:42.079371 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079324 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:58:42.079371 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079334 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:58:42.079371 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.079341 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:58:42.080660 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.080650 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:58:42.080706 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.080663 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:58:42.084017 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.083997 2576 server.go:1268] "Started kubelet" Jul 15 19:58:42.084151 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.084114 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:58:42.084229 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.084175 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:58:42.084283 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.084258 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:58:42.084984 ip-10-0-141-87 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:58:42.085557 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.085506 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:58:42.087966 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.087266 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:58:42.088274 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.088249 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:58:42.088487 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.088451 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:58:42.088584 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.088528 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:58:42.088654 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.088614 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:58:42.093552 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.089733 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-87.ec2.internal.18c28ee6476b3015 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-87.ec2.internal,UID:ip-10-0-141-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-87.ec2.internal,},FirstTimestamp:2026-07-15 19:58:42.083966997 +0000 UTC m=+0.354716490,LastTimestamp:2026-07-15 19:58:42.083966997 +0000 UTC m=+0.354716490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-87.ec2.internal,}" Jul 15 19:58:42.094451 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.094428 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:58:42.094451 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.094435 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:58:42.094901 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.094681 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.095135 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095115 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:58:42.095135 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095115 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:58:42.095275 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095148 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:58:42.095315 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095298 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:58:42.095315 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095306 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:58:42.095767 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095748 2576 factory.go:153] Registering CRI-O factory Jul 15 19:58:42.095867 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095771 2576 factory.go:223] Registration of the crio container factory successfully Jul 15 19:58:42.095867 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095847 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:58:42.095867 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095858 2576 factory.go:55] Registering systemd factory Jul 15 19:58:42.095867 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095864 2576 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:58:42.096061 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095882 2576 factory.go:103] Registering Raw factory Jul 15 19:58:42.096061 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.095891 2576 manager.go:1196] Started watching for new ooms in manager Jul 15 19:58:42.096560 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.096538 2576 manager.go:349] Starting recovery of all containers Jul 15 19:58:42.101426 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.101397 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 19:58:42.101595 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.101562 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 19:58:42.103763 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.103720 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:58:42.106486 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.106460 2576 manager.go:354] Recovery completed Jul 15 19:58:42.108080 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.107880 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrdpt" Jul 15 19:58:42.111964 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.111950 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.113868 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.113818 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.113868 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.113865 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.113996 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.113876 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.114429 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.114415 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:58:42.114429 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.114427 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:58:42.114540 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.114444 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.114540 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.114446 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:58:42.115261 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115245 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.115317 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115271 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.115317 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115281 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.115794 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115782 2576 policy_none.go:49] "None policy: Start" Jul 15 19:58:42.115838 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115801 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:58:42.115838 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.115815 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:58:42.116159 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.116086 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-87.ec2.internal.18c28ee6493329fe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-87.ec2.internal,UID:ip-10-0-141-87.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-87.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-87.ec2.internal,},FirstTimestamp:2026-07-15 19:58:42.113849854 +0000 UTC m=+0.384599346,LastTimestamp:2026-07-15 19:58:42.113849854 +0000 UTC m=+0.384599346,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-87.ec2.internal,}" Jul 15 19:58:42.116910 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.116894 2576 policy_none.go:47] "Start" Jul 15 19:58:42.117809 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.117791 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrdpt" Jul 15 19:58:42.163573 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.163556 2576 manager.go:337] "Starting Device Plugin manager" Jul 15 19:58:42.163669 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.163592 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:58:42.163669 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.163602 2576 server.go:87] "Starting device plugin registration server" Jul 15 19:58:42.163906 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.163859 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:58:42.163906 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.163880 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:58:42.164030 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.163964 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:58:42.164090 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.164042 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:58:42.164090 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.164062 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:58:42.164586 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.164562 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:58:42.164656 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.164616 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.264121 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.264026 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.265304 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.265084 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.265304 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.265114 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.265304 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.265129 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.265304 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.265160 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.273563 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.273538 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.273686 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.273572 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-87.ec2.internal\": node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.280334 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.280306 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:58:42.280447 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.280352 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:58:42.280447 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.280375 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:58:42.280447 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.280412 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:58:42.281935 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.281914 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:42.284272 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.284256 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.381362 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.381323 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"] Jul 15 19:58:42.381486 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.381404 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.382450 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.382435 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.382503 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.382469 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.382503 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.382479 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.383868 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.383855 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.383932 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.383912 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.383982 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.383941 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.384331 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.384316 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.384659 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384635 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.384760 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384666 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.384760 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384676 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.384760 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384638 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.384760 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384734 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.384760 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.384748 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.386078 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.386063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.386145 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.386089 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:58:42.386838 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.386808 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:58:42.386928 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.386855 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:58:42.386928 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.386870 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:58:42.397611 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.397582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.397741 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.397618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.397741 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.397639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/152de02e452d1b71aa2bf2bbb50373e3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"152de02e452d1b71aa2bf2bbb50373e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.418838 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.418811 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-87.ec2.internal\" not found" node="ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.422363 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.422342 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-87.ec2.internal\" not found" node="ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.484858 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.484810 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.498736 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.498881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.498881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/152de02e452d1b71aa2bf2bbb50373e3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"152de02e452d1b71aa2bf2bbb50373e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.498881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.498881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498858 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ecccf2c5859664e97009bf871dc64d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"ecccf2c5859664e97009bf871dc64d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.498881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.498841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/152de02e452d1b71aa2bf2bbb50373e3-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"152de02e452d1b71aa2bf2bbb50373e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.585264 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.585189 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.685759 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.685713 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.720293 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.720259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.725459 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.725436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:42.785874 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.785814 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.886460 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:42.886397 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found" Jul 15 19:58:42.894631 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.894610 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:42.995068 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:42.995036 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" Jul 15 19:58:43.005886 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.005864 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:58:43.009010 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.008990 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" Jul 15 19:58:43.020076 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.020039 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:58:43.038240 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.038207 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:58:43.038396 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.038354 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:58:43.038448 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.038383 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:58:43.074556 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.074519 2576 apiserver.go:52] "Watching apiserver" Jul 15 19:58:43.089585 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.089561 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:58:43.089985 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.089958 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-8d2rf","kube-system/konnectivity-agent-7h9pp","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9sgff","openshift-image-registry/node-ca-6clq5","openshift-multus/multus-r22jf","openshift-ovn-kubernetes/ovnkube-node-dkmdg","kube-system/global-pull-secret-syncer-dfgv2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72","openshift-dns/node-resolver-tx46d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","openshift-multus/multus-additional-cni-plugins-4s5fc","openshift-multus/network-metrics-daemon-2vx4g","openshift-network-diagnostics/network-check-target-765p2"] Jul 15 19:58:43.092013 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.091992 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.094220 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.093789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.094220 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.093950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:43.095318 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095289 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:58:43.095532 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:58:43.095597 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095544 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:58:43.095762 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095720 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:58:43.096002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095807 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hq6xn\"" Jul 15 19:58:43.096002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095873 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.096002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095919 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:58:43.096287 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.095873 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.096403 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.096385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:43.096510 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.096472 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:43.097785 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.097769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.099423 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.099405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.100090 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.100069 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.100232 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.100146 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.100864 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.100850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zj8ql\"" Jul 15 19:58:43.101139 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.101127 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.102387 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-kubelet\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102479 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102402 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-systemd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102479 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.102586 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-config\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102586 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/069a64e5-5074-4274-8b9e-020eeb44415a-ovn-node-metrics-cert\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102586 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw99p\" (UniqueName: \"kubernetes.io/projected/069a64e5-5074-4274-8b9e-020eeb44415a-kube-api-access-tw99p\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102729 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-systemd-units\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102729 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102729 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-log-socket\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102729 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-netd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102887 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-env-overrides\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102887 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102777 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.102949 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-slash\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.102949 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-netns\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103006 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-var-lib-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103006 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.102978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-ovn\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103006 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-bin\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103114 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103114 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:43.103114 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-etc-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-node-log\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-script-lib\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.103251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spj5k\" (UniqueName: \"kubernetes.io/projected/05151441-6723-414d-85ce-569fa23139d7-kube-api-access-spj5k\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.104092 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.104092 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103778 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.104092 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103778 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:58:43.104092 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.103985 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.104384 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.104368 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m6wcr\"" Jul 15 19:58:43.104448 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.104396 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.104917 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.104887 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105107 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105126 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105140 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tmjg\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105190 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c6d6v\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105140 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.105380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105129 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:58:43.105640 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.105476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:58:43.106869 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.106851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.107342 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.107259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.108210 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108191 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7csg\"" Jul 15 19:58:43.108449 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:58:43.108518 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.108784 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108540 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hmmcg\"" Jul 15 19:58:43.108784 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:58:43.108784 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:58:43.108964 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.108850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.108964 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.108912 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:43.109758 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.109738 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:58:43.109881 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.109862 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rj857\"" Jul 15 19:58:43.109988 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.109973 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:58:43.110396 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.110379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.117383 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.117360 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:58:43.117515 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.117415 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:58:43.117515 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.117360 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9lbbx\"" Jul 15 19:58:43.117515 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.117453 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:58:43.120727 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.120682 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 19:53:43 +0000 UTC" deadline="2028-03-20 13:23:21.621671737 +0000 UTC" Jul 15 19:58:43.120727 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.120726 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14729h24m38.500949351s" Jul 15 19:58:43.136199 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.136165 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ccwjz" Jul 15 19:58:43.145183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.145115 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ccwjz" Jul 15 19:58:43.195867 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.195842 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:58:43.203890 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-kubelet\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.203995 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-systemd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.203995 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/069a64e5-5074-4274-8b9e-020eeb44415a-ovn-node-metrics-cert\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.203995 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-multus\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.203995 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-systemd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.203986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-kubelet\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-kubelet-config\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-serviceca\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-systemd-units\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-netd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-env-overrides\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204175 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-kubelet\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-systemd-units\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-netd\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a7fb83e-b912-4905-ad30-1a91cc79dd94-iptables-alerter-script\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78e336f2-a47f-4429-abab-4d52971db61b-agent-certs\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-k8s-cni-cncf-io\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204305 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-multus-certs\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204402 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a7fb83e-b912-4905-ad30-1a91cc79dd94-host-slash\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-systemd\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-hostroot\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysconfig\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-lib-modules\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-ovn-kubernetes\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-etc-selinux\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-sys-fs\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-env-overrides\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-os-release\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-netns\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.204790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-socket-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-cni-binary-copy\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-socket-dir-parent\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-os-release\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.204976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-slash\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-slash\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-netns\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-run-netns\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-var-lib-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnq77\" (UniqueName: \"kubernetes.io/projected/8fee5a20-8480-4e96-9729-a037774ee983-kube-api-access-hnq77\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-var-lib-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.205084 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6bh\" (UniqueName: \"kubernetes.io/projected/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-kube-api-access-gj6bh\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.205375 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.205217 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:43.705199504 +0000 UTC m=+1.975948984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-tmp\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-host\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-bin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-var-lib-kubelet\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnw8\" (UniqueName: \"kubernetes.io/projected/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kube-api-access-wlnw8\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-config\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw99p\" (UniqueName: \"kubernetes.io/projected/069a64e5-5074-4274-8b9e-020eeb44415a-kube-api-access-tw99p\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kh6d\" (UniqueName: \"kubernetes.io/projected/6a7fb83e-b912-4905-ad30-1a91cc79dd94-kube-api-access-2kh6d\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-host\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205449 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-etc-tuned\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjvr\" (UniqueName: \"kubernetes.io/projected/1db47235-8805-4bc2-b623-7a103cc016fa-kube-api-access-9bjvr\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78e336f2-a47f-4429-abab-4d52971db61b-konnectivity-ca\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-hosts-file\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.206221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-tmp-dir\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-log-socket\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205720 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-modprobe-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205743 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-run\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-log-socket\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-ovn\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-bin\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-run-ovn\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-node-log\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-script-lib\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205894 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-config\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-node-log\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.205979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-host-cni-bin\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-cnibin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.206774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-conf-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-cnibin\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-device-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-etc-kubernetes\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-dbus\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-conf\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-sys\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-kubernetes\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tkn\" (UniqueName: \"kubernetes.io/projected/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-kube-api-access-84tkn\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-registration-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/069a64e5-5074-4274-8b9e-020eeb44415a-ovnkube-script-lib\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-etc-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spj5k\" (UniqueName: \"kubernetes.io/projected/05151441-6723-414d-85ce-569fa23139d7-kube-api-access-spj5k\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-system-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-multus-daemon-config\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbht\" (UniqueName: \"kubernetes.io/projected/79b57609-b496-4140-9565-5b91749c53f0-kube-api-access-xqbht\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.207260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206437 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/069a64e5-5074-4274-8b9e-020eeb44415a-etc-openvswitch\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.207898 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.206472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.207898 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.207569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/069a64e5-5074-4274-8b9e-020eeb44415a-ovn-node-metrics-cert\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.210793 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.210754 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152de02e452d1b71aa2bf2bbb50373e3.slice/crio-f44d726393eec783f327fc5576054128c5d43c7d62b2cc9935c2f90588f19b71 WatchSource:0}: Error finding container f44d726393eec783f327fc5576054128c5d43c7d62b2cc9935c2f90588f19b71: Status 404 returned error can't find the container with id f44d726393eec783f327fc5576054128c5d43c7d62b2cc9935c2f90588f19b71 Jul 15 19:58:43.211104 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.211081 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecccf2c5859664e97009bf871dc64d68.slice/crio-7fcaaff49dafed005bd0ce8f042752560601742172c21c1a2229d9786209c6cf WatchSource:0}: Error finding container 7fcaaff49dafed005bd0ce8f042752560601742172c21c1a2229d9786209c6cf: Status 404 returned error can't find the container with id 7fcaaff49dafed005bd0ce8f042752560601742172c21c1a2229d9786209c6cf Jul 15 19:58:43.212156 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.212138 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:58:43.212253 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.212166 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:58:43.212253 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.212181 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:43.212253 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.212246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:43.712227313 +0000 UTC m=+1.982976805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:43.215099 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.214966 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw99p\" (UniqueName: \"kubernetes.io/projected/069a64e5-5074-4274-8b9e-020eeb44415a-kube-api-access-tw99p\") pod \"ovnkube-node-dkmdg\" (UID: \"069a64e5-5074-4274-8b9e-020eeb44415a\") " pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.215350 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.215325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spj5k\" (UniqueName: \"kubernetes.io/projected/05151441-6723-414d-85ce-569fa23139d7-kube-api-access-spj5k\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:43.215461 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.215447 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:58:43.283551 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.283493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"152de02e452d1b71aa2bf2bbb50373e3","Type":"ContainerStarted","Data":"f44d726393eec783f327fc5576054128c5d43c7d62b2cc9935c2f90588f19b71"} Jul 15 19:58:43.284637 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.284616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"ecccf2c5859664e97009bf871dc64d68","Type":"ContainerStarted","Data":"7fcaaff49dafed005bd0ce8f042752560601742172c21c1a2229d9786209c6cf"} Jul 15 19:58:43.306737 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a7fb83e-b912-4905-ad30-1a91cc79dd94-host-slash\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.306737 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-systemd\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-hostroot\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-hostroot\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a7fb83e-b912-4905-ad30-1a91cc79dd94-host-slash\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-systemd\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysconfig\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-lib-modules\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306898 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysconfig\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-etc-selinux\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-sys-fs\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-os-release\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.306962 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-netns\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-sys-fs\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.306998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-netns\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-etc-selinux\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307030 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-os-release\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-lib-modules\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-socket-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-cni-binary-copy\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-socket-dir-parent\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-os-release\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-socket-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnq77\" (UniqueName: \"kubernetes.io/projected/8fee5a20-8480-4e96-9729-a037774ee983-kube-api-access-hnq77\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.307310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-socket-dir-parent\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6bh\" (UniqueName: \"kubernetes.io/projected/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-kube-api-access-gj6bh\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-tmp\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-os-release\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-host\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-bin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-system-cni-dir\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-var-lib-kubelet\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnw8\" (UniqueName: \"kubernetes.io/projected/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kube-api-access-wlnw8\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kh6d\" (UniqueName: \"kubernetes.io/projected/6a7fb83e-b912-4905-ad30-1a91cc79dd94-kube-api-access-2kh6d\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-host\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-etc-tuned\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-bin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-var-lib-kubelet\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjvr\" (UniqueName: \"kubernetes.io/projected/1db47235-8805-4bc2-b623-7a103cc016fa-kube-api-access-9bjvr\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78e336f2-a47f-4429-abab-4d52971db61b-konnectivity-ca\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-hosts-file\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-tmp-dir\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-modprobe-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-run\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-modprobe-d\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-cnibin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-conf-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.307925 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-cnibin\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-host\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-hosts-file\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:43.307974 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:43.807957766 +0000 UTC m=+2.078707263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:43.308908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-tmp-dir\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-run\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-conf-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-cnibin\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-multus-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.307926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79b57609-b496-4140-9565-5b91749c53f0-cnibin\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-device-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-device-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-host\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-etc-kubernetes\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-etc-kubernetes\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-dbus\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/78e336f2-a47f-4429-abab-4d52971db61b-konnectivity-ca\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-conf\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-sys\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-dbus\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-kubernetes\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-sys\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.309718 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tkn\" (UniqueName: \"kubernetes.io/projected/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-kube-api-access-84tkn\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79b57609-b496-4140-9565-5b91749c53f0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-registration-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-sysctl-conf\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db47235-8805-4bc2-b623-7a103cc016fa-etc-kubernetes\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-registration-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-system-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-multus-daemon-config\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-system-cni-dir\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbht\" (UniqueName: \"kubernetes.io/projected/79b57609-b496-4140-9565-5b91749c53f0-kube-api-access-xqbht\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-multus\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-kubelet-config\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-serviceca\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-kubelet\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a7fb83e-b912-4905-ad30-1a91cc79dd94-iptables-alerter-script\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.310276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-kubelet\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.308997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-var-lib-cni-multus\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78e336f2-a47f-4429-abab-4d52971db61b-agent-certs\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ccd8254-4a7f-43c7-aee8-bd36cd411958-kubelet-config\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-k8s-cni-cncf-io\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-k8s-cni-cncf-io\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-multus-certs\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8fee5a20-8480-4e96-9729-a037774ee983-host-run-multus-certs\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309240 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-multus-daemon-config\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fee5a20-8480-4e96-9729-a037774ee983-cni-binary-copy\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-serviceca\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.309532 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6a7fb83e-b912-4905-ad30-1a91cc79dd94-iptables-alerter-script\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.310651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-etc-tuned\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.310873 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.310783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db47235-8805-4bc2-b623-7a103cc016fa-tmp\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.311358 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.311342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/78e336f2-a47f-4429-abab-4d52971db61b-agent-certs\") pod \"konnectivity-agent-7h9pp\" (UID: \"78e336f2-a47f-4429-abab-4d52971db61b\") " pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.319818 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.319783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6bh\" (UniqueName: \"kubernetes.io/projected/8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd-kube-api-access-gj6bh\") pod \"node-resolver-tx46d\" (UID: \"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd\") " pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.319994 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.319871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kh6d\" (UniqueName: \"kubernetes.io/projected/6a7fb83e-b912-4905-ad30-1a91cc79dd94-kube-api-access-2kh6d\") pod \"iptables-alerter-8d2rf\" (UID: \"6a7fb83e-b912-4905-ad30-1a91cc79dd94\") " pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.320098 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.320075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnw8\" (UniqueName: \"kubernetes.io/projected/c393f5cd-50c2-4e99-a59c-3e4f44a8dd43-kube-api-access-wlnw8\") pod \"aws-ebs-csi-driver-node-9zh72\" (UID: \"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.320704 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.320684 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnq77\" (UniqueName: \"kubernetes.io/projected/8fee5a20-8480-4e96-9729-a037774ee983-kube-api-access-hnq77\") pod \"multus-r22jf\" (UID: \"8fee5a20-8480-4e96-9729-a037774ee983\") " pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.321122 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.321103 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tkn\" (UniqueName: \"kubernetes.io/projected/bf7ab018-02e3-4dd9-bd83-26c6719e02c7-kube-api-access-84tkn\") pod \"node-ca-6clq5\" (UID: \"bf7ab018-02e3-4dd9-bd83-26c6719e02c7\") " pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.321314 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.321298 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbht\" (UniqueName: \"kubernetes.io/projected/79b57609-b496-4140-9565-5b91749c53f0-kube-api-access-xqbht\") pod \"multus-additional-cni-plugins-4s5fc\" (UID: \"79b57609-b496-4140-9565-5b91749c53f0\") " pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.321382 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.321366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjvr\" (UniqueName: \"kubernetes.io/projected/1db47235-8805-4bc2-b623-7a103cc016fa-kube-api-access-9bjvr\") pod \"tuned-9sgff\" (UID: \"1db47235-8805-4bc2-b623-7a103cc016fa\") " pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.342623 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.342593 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:43.399751 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.399667 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:43.412078 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.412032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:58:43.421191 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.421157 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069a64e5_5074_4274_8b9e_020eeb44415a.slice/crio-7fc65f5be049ce49ca9fb4aa8ac88e577187e24b7d03102ac962eb86df9dc88f WatchSource:0}: Error finding container 7fc65f5be049ce49ca9fb4aa8ac88e577187e24b7d03102ac962eb86df9dc88f: Status 404 returned error can't find the container with id 7fc65f5be049ce49ca9fb4aa8ac88e577187e24b7d03102ac962eb86df9dc88f Jul 15 19:58:43.428594 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.428570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9sgff" Jul 15 19:58:43.436280 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.436250 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db47235_8805_4bc2_b623_7a103cc016fa.slice/crio-4e0cd7294a2a2d34eda7634d739d235042c43daeb761de9cc1b8ac42765b723a WatchSource:0}: Error finding container 4e0cd7294a2a2d34eda7634d739d235042c43daeb761de9cc1b8ac42765b723a: Status 404 returned error can't find the container with id 4e0cd7294a2a2d34eda7634d739d235042c43daeb761de9cc1b8ac42765b723a Jul 15 19:58:43.442484 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.442459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6clq5" Jul 15 19:58:43.448644 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.448614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r22jf" Jul 15 19:58:43.451453 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.451425 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7ab018_02e3_4dd9_bd83_26c6719e02c7.slice/crio-d83c9accb60c201e2939ae8a433c9d3d699d80b7e631928a0b1891ac4f981317 WatchSource:0}: Error finding container d83c9accb60c201e2939ae8a433c9d3d699d80b7e631928a0b1891ac4f981317: Status 404 returned error can't find the container with id d83c9accb60c201e2939ae8a433c9d3d699d80b7e631928a0b1891ac4f981317 Jul 15 19:58:43.454899 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.454876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" Jul 15 19:58:43.457184 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.457158 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fee5a20_8480_4e96_9729_a037774ee983.slice/crio-2a93ba0c82a810d268c33aa582210285b46a000eed3dbd30a64c50894796bee9 WatchSource:0}: Error finding container 2a93ba0c82a810d268c33aa582210285b46a000eed3dbd30a64c50894796bee9: Status 404 returned error can't find the container with id 2a93ba0c82a810d268c33aa582210285b46a000eed3dbd30a64c50894796bee9 Jul 15 19:58:43.463408 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.463383 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc393f5cd_50c2_4e99_a59c_3e4f44a8dd43.slice/crio-af466c35f767c6a72db239402477be841e7770c23a8e9f222d85f9f739f0abe1 WatchSource:0}: Error finding container af466c35f767c6a72db239402477be841e7770c23a8e9f222d85f9f739f0abe1: Status 404 returned error can't find the container with id af466c35f767c6a72db239402477be841e7770c23a8e9f222d85f9f739f0abe1 Jul 15 19:58:43.474205 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.474178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8d2rf" Jul 15 19:58:43.482240 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.482215 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7fb83e_b912_4905_ad30_1a91cc79dd94.slice/crio-793ea7aadc609908f399b4830c3a5a6547aebf1eb8d17b3bef82ff40b2057695 WatchSource:0}: Error finding container 793ea7aadc609908f399b4830c3a5a6547aebf1eb8d17b3bef82ff40b2057695: Status 404 returned error can't find the container with id 793ea7aadc609908f399b4830c3a5a6547aebf1eb8d17b3bef82ff40b2057695 Jul 15 19:58:43.514885 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.514808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:58:43.520934 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.520906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tx46d" Jul 15 19:58:43.522472 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.522445 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e336f2_a47f_4429_abab_4d52971db61b.slice/crio-3744acb82e787a54eb94fa241fbbfb3c51649bc580123a7519855e0437eb860d WatchSource:0}: Error finding container 3744acb82e787a54eb94fa241fbbfb3c51649bc580123a7519855e0437eb860d: Status 404 returned error can't find the container with id 3744acb82e787a54eb94fa241fbbfb3c51649bc580123a7519855e0437eb860d Jul 15 19:58:43.529071 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:43.528677 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" Jul 15 19:58:43.529739 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.529720 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c00e0e0_0d9e_4d41_a987_b0d7a8dc51cd.slice/crio-2671545816dd5941f2ba1f2cdb0a10696a284a6b369c7eb5055164a5d6e99224 WatchSource:0}: Error finding container 2671545816dd5941f2ba1f2cdb0a10696a284a6b369c7eb5055164a5d6e99224: Status 404 returned error can't find the container with id 2671545816dd5941f2ba1f2cdb0a10696a284a6b369c7eb5055164a5d6e99224 Jul 15 19:58:43.537684 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:58:43.537658 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b57609_b496_4140_9565_5b91749c53f0.slice/crio-563fafb299ca68d30784bc6324d68f99bac194dd3256e10b209a821c1056418e WatchSource:0}: Error finding container 563fafb299ca68d30784bc6324d68f99bac194dd3256e10b209a821c1056418e: Status 404 returned error can't find the container with id 563fafb299ca68d30784bc6324d68f99bac194dd3256e10b209a821c1056418e Jul 15 19:58:45.274406 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.274317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:45.274406 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.274370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274460 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274484 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274501 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274515 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274525 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:46.274509735 +0000 UTC m=+2.984075968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:45.274582 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.274565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:46.274554107 +0000 UTC m=+2.984120340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:45.375639 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.375295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:45.375639 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.375507 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:45.375639 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.375575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:46.375556619 +0000 UTC m=+3.085122864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:45.709424 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.709305 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:53:44 +0000 UTC" deadline="2028-03-23 17:32:07.559076266 +0000 UTC" Jul 15 19:58:45.709424 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.709370 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14805h33m21.849710782s" Jul 15 19:58:45.736506 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.736260 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:45.843081 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.842519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:45.843081 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:45.842679 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:45.870739 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.870606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerStarted","Data":"563fafb299ca68d30784bc6324d68f99bac194dd3256e10b209a821c1056418e"} Jul 15 19:58:45.873360 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.873226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tx46d" event={"ID":"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd","Type":"ContainerStarted","Data":"2671545816dd5941f2ba1f2cdb0a10696a284a6b369c7eb5055164a5d6e99224"} Jul 15 19:58:45.886363 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.886326 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7h9pp" event={"ID":"78e336f2-a47f-4429-abab-4d52971db61b","Type":"ContainerStarted","Data":"3744acb82e787a54eb94fa241fbbfb3c51649bc580123a7519855e0437eb860d"} Jul 15 19:58:45.891964 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.891905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8d2rf" event={"ID":"6a7fb83e-b912-4905-ad30-1a91cc79dd94","Type":"ContainerStarted","Data":"793ea7aadc609908f399b4830c3a5a6547aebf1eb8d17b3bef82ff40b2057695"} Jul 15 19:58:45.925004 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.924941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" event={"ID":"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43","Type":"ContainerStarted","Data":"af466c35f767c6a72db239402477be841e7770c23a8e9f222d85f9f739f0abe1"} Jul 15 19:58:45.934157 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.934113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6clq5" event={"ID":"bf7ab018-02e3-4dd9-bd83-26c6719e02c7","Type":"ContainerStarted","Data":"d83c9accb60c201e2939ae8a433c9d3d699d80b7e631928a0b1891ac4f981317"} Jul 15 19:58:45.947064 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.947024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r22jf" event={"ID":"8fee5a20-8480-4e96-9729-a037774ee983","Type":"ContainerStarted","Data":"2a93ba0c82a810d268c33aa582210285b46a000eed3dbd30a64c50894796bee9"} Jul 15 19:58:45.959083 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.958956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9sgff" event={"ID":"1db47235-8805-4bc2-b623-7a103cc016fa","Type":"ContainerStarted","Data":"4e0cd7294a2a2d34eda7634d739d235042c43daeb761de9cc1b8ac42765b723a"} Jul 15 19:58:45.971458 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:45.971366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"7fc65f5be049ce49ca9fb4aa8ac88e577187e24b7d03102ac962eb86df9dc88f"} Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.281955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.282053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282187 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:48.282229357 +0000 UTC m=+4.991795588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282661 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282681 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282695 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:46.282834 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.282744 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:48.282728302 +0000 UTC m=+4.992294535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:46.383115 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.383078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:46.383307 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.383262 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:46.383382 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.383334 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:48.383313001 +0000 UTC m=+5.092879234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:46.710802 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.710444 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:53:44 +0000 UTC" deadline="2028-02-18 10:07:09.791089513 +0000 UTC" Jul 15 19:58:46.710802 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.710483 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13982h8m23.080610772s" Jul 15 19:58:46.843226 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.842674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:46.843226 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.842811 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:46.843472 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:46.843303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:46.843472 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:46.843403 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:47.619726 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:47.619697 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:58:47.853997 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:47.853894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:47.854474 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:47.854145 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:48.299725 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:48.299633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:48.299725 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:48.299714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:48.299945 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.299869 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:58:48.299945 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.299890 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:58:48.299945 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.299903 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:48.300122 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.299966 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:48.300122 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.299982 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:52.299963951 +0000 UTC m=+9.009530183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:48.300122 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.300052 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:52.300031849 +0000 UTC m=+9.009598094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:48.400673 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:48.400340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:48.400673 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.400538 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:48.400673 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.400603 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:58:52.400584176 +0000 UTC m=+9.110150410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:48.842837 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:48.842400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:48.842837 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.842541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:48.842837 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:48.842725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:48.842837 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:48.842806 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:49.842556 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:49.842525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:49.842935 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:49.842676 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:50.842200 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:50.842136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:50.842200 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:50.842175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:50.842510 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:50.842282 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:50.842510 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:50.842431 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:51.842537 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:51.841949 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:51.842537 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:51.842129 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:52.335139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:52.335213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335394 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335413 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335427 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335492 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:00.335470135 +0000 UTC m=+17.045036416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335886 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:52.336033 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.335933 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:00.335918861 +0000 UTC m=+17.045485098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:58:52.435892 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:52.435853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:52.436102 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.436063 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:52.436154 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.436142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:00.436119734 +0000 UTC m=+17.145685987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:58:52.842075 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:52.841999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:52.842272 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.842153 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:52.842272 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:52.842211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:52.842389 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:52.842324 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:53.843187 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:53.842922 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:53.843666 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:53.843304 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:54.842191 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:54.842093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:54.842191 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:54.842114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:54.842461 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:54.842230 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:54.842461 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:54.842286 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:55.842240 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:55.842198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:55.842688 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:55.842347 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:56.842171 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:56.842134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:56.842353 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:56.842134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:56.842353 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:56.842261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:56.842721 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:56.842392 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:57.842494 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:57.842454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:57.842902 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:57.842582 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:58:58.842734 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:58.842694 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:58:58.843204 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:58.842749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:58:58.843204 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:58.842835 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:58:58.843204 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:58.842973 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:58:59.842326 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:58:59.842286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:58:59.842526 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:58:59.842446 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:00.394548 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:00.394512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:00.394570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394688 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394702 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394719 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394728 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394757 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:16.394739592 +0000 UTC m=+33.104305825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:59:00.395002 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.394777 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:16.394767562 +0000 UTC m=+33.104333794 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:59:00.495777 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:00.495732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:00.495958 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.495887 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:59:00.496031 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.495970 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:16.495949789 +0000 UTC m=+33.205516036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:59:00.842723 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:00.842683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:00.842723 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:00.842722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:00.842943 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.842827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:00.842943 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:00.842919 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:01.841984 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:01.841939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:01.842561 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:01.842118 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:02.842548 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:02.842509 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:02.843042 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:02.842547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:02.843042 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:02.842642 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:02.843042 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:02.842777 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:03.843333 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:03.843297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:03.843798 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:03.843430 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:04.842611 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:04.842569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:04.842809 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:04.842717 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:04.843151 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:04.842569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:04.843291 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:04.843255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:05.842750 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:05.842170 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:05.842750 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:05.842434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:06.027558 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.027263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"152de02e452d1b71aa2bf2bbb50373e3","Type":"ContainerStarted","Data":"d1819ae9273421ddb64f59a5e766676300556956ca9dfc0f72446a460ac115bf"} Jul 15 19:59:06.028810 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.028783 2576 generic.go:356] "Generic (PLEG): container finished" podID="ecccf2c5859664e97009bf871dc64d68" containerID="1a0f7c94721d129237aa965f2bca509ef4ddf0ece776e449c359b43be32797c8" exitCode=0 Jul 15 19:59:06.028940 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.028873 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"ecccf2c5859664e97009bf871dc64d68","Type":"ContainerDied","Data":"1a0f7c94721d129237aa965f2bca509ef4ddf0ece776e449c359b43be32797c8"} Jul 15 19:59:06.030584 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.030558 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="0a6d48d39c71a3cdb3608fcd7873e65cce305129c6245169aceb36b0fb66f875" exitCode=0 Jul 15 19:59:06.030687 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.030626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"0a6d48d39c71a3cdb3608fcd7873e65cce305129c6245169aceb36b0fb66f875"} Jul 15 19:59:06.032173 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.032150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tx46d" event={"ID":"8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd","Type":"ContainerStarted","Data":"da551a64f32c63e6df55f3e7409fa1398469f1130c27dcff8e8206bc9770bf98"} Jul 15 19:59:06.033622 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.033602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7h9pp" event={"ID":"78e336f2-a47f-4429-abab-4d52971db61b","Type":"ContainerStarted","Data":"3b97dc2ab576b6c39dbe6fc561ba55e8249450e867d9ba9087ee48e1270ab20a"} Jul 15 19:59:06.034915 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.034896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" event={"ID":"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43","Type":"ContainerStarted","Data":"321cb0aa4c8a16dcb8aadd541e2c2f82b15ee2a3419a443abbc3636bd5f684a9"} Jul 15 19:59:06.036349 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.036328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6clq5" event={"ID":"bf7ab018-02e3-4dd9-bd83-26c6719e02c7","Type":"ContainerStarted","Data":"cd2bf6ec354f28e203163f4be15c96e5058fb710d6f9f3f4093170c9075d692d"} Jul 15 19:59:06.037641 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.037620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r22jf" event={"ID":"8fee5a20-8480-4e96-9729-a037774ee983","Type":"ContainerStarted","Data":"cb8577085f92df26963d4e7254430b08df96f0b366118c4780cf983faa1fb3ac"} Jul 15 19:59:06.038920 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.038899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9sgff" event={"ID":"1db47235-8805-4bc2-b623-7a103cc016fa","Type":"ContainerStarted","Data":"3e33828aaf4d968c158bf5f68c1aa84bd9c0c2f0525b348885e165741e4b55fe"} Jul 15 19:59:06.041447 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041430 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 19:59:06.041761 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041741 2576 generic.go:356] "Generic (PLEG): container finished" podID="069a64e5-5074-4274-8b9e-020eeb44415a" containerID="13bfd0a3638a437b29a8ec1548cb92e7bbb8e189856515b3934659c5e4820caa" exitCode=1 Jul 15 19:59:06.041808 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041771 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"691665dff8dd63ce46f9b1ab47129f2a1254c523d56ebfa64a83aca6db7723a2"} Jul 15 19:59:06.041842 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"aee5e3d98004e03c43e45d9c5381c84510ad1d7ab374873706d7bb1182e3305d"} Jul 15 19:59:06.041876 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"e95d9474818519556430ff67001eab182c8cf2135de6606c17fe7cca8da7e465"} Jul 15 19:59:06.041876 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"8caf0c5bc03a26d2d4fdcfa6d1449644c19e0c082344ce7b60728d1d74eeb9ff"} Jul 15 19:59:06.041935 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerDied","Data":"13bfd0a3638a437b29a8ec1548cb92e7bbb8e189856515b3934659c5e4820caa"} Jul 15 19:59:06.041935 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.041924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"6aee403a81fb6179f1dc766e8e94d54d4f98b496040952c08532c760aa9d9769"} Jul 15 19:59:06.050561 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.050482 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" podStartSLOduration=22.050468387 podStartE2EDuration="22.050468387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.049941282 +0000 UTC m=+22.759507534" watchObservedRunningTime="2026-07-15 19:59:06.050468387 +0000 UTC m=+22.760034638" Jul 15 19:59:06.084761 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.084707 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6clq5" podStartSLOduration=2.932196804 podStartE2EDuration="23.084691208s" totalImagesPullingTime="20.152494404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.084631791 +0000 UTC m=+22.794198046" watchObservedRunningTime="2026-07-15 19:59:06.084691208 +0000 UTC m=+22.794257459" Jul 15 19:59:06.133075 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.133025 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7h9pp" podStartSLOduration=3.060264098 podStartE2EDuration="23.132987869s" totalImagesPullingTime="20.072723771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.106080953 +0000 UTC m=+22.815647202" watchObservedRunningTime="2026-07-15 19:59:06.132987869 +0000 UTC m=+22.842554120" Jul 15 19:59:06.133503 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.133478 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tx46d" podStartSLOduration=3.069436587 podStartE2EDuration="23.133470222s" totalImagesPullingTime="20.064033635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.133153179 +0000 UTC m=+22.842719415" watchObservedRunningTime="2026-07-15 19:59:06.133470222 +0000 UTC m=+22.843036473" Jul 15 19:59:06.212817 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.212708 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9sgff" podStartSLOduration=3.0559522709999998 podStartE2EDuration="23.212693223s" totalImagesPullingTime="20.156740952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.212231212 +0000 UTC m=+22.921797464" watchObservedRunningTime="2026-07-15 19:59:06.212693223 +0000 UTC m=+22.922259474" Jul 15 19:59:06.232599 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.232557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:06.244189 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.244135 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r22jf" podStartSLOduration=3.045568144 podStartE2EDuration="23.24411714s" totalImagesPullingTime="20.198548996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:06.240957122 +0000 UTC m=+22.950523374" watchObservedRunningTime="2026-07-15 19:59:06.24411714 +0000 UTC m=+22.953683391" Jul 15 19:59:06.363513 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.363477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:06.506289 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.506195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:06.617379 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.617330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:06.736554 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.736507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:06.842112 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.842077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:06.842291 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.842077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:06.842291 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:06.842205 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:06.842291 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:06.842260 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:06.873908 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:06.873875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:07.046768 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.046728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"ecccf2c5859664e97009bf871dc64d68","Type":"ContainerStarted","Data":"491c8b421ed13531e2fd0a264d9226839e14eaa62a9a84ca5f41e6576e45409d"} Jul 15 19:59:07.049221 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.048939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8d2rf" event={"ID":"6a7fb83e-b912-4905-ad30-1a91cc79dd94","Type":"ContainerStarted","Data":"a2557df0e2b8f26c358b1fc25d2ee9605e8a71b14be0b714fc6778e2bc7f24e0"} Jul 15 19:59:07.073671 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.073622 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" podStartSLOduration=23.073606005 podStartE2EDuration="23.073606005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:07.073271488 +0000 UTC m=+23.782837740" watchObservedRunningTime="2026-07-15 19:59:07.073606005 +0000 UTC m=+23.783172257" Jul 15 19:59:07.120429 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.120358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8d2rf" podStartSLOduration=3.994419818 podStartE2EDuration="24.120336893s" totalImagesPullingTime="20.125917075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:07.120158618 +0000 UTC m=+23.829724870" watchObservedRunningTime="2026-07-15 19:59:07.120336893 +0000 UTC m=+23.829903145" Jul 15 19:59:07.128160 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.128126 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:59:07.271241 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.271182 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:07.428207 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.428127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:07.744541 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.744313 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:59:07.128155057Z","UUID":"a243f199-e1ce-4958-a923-ef18eccc67e1","Handler":null,"Name":"","Endpoint":""} Jul 15 19:59:07.746324 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.746302 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:59:07.746466 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.746346 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:59:07.842605 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.842568 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:07.842771 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:07.842748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:07.881805 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:07.881604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:08.054180 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.054083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" event={"ID":"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43","Type":"ContainerStarted","Data":"5f05abda8817cdc15a65d54b993ea0a773369ce7249b58776a994fd06ddaecc8"} Jul 15 19:59:08.076255 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.076207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:08.339232 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.339206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:59:08.339792 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.339766 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:59:08.841767 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.841742 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:08.841915 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:08.841860 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:08.841956 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.841921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:08.842073 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:08.842051 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:08.943563 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.943523 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:59:08.944101 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:08.944086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7h9pp" Jul 15 19:59:09.060080 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:09.060049 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 19:59:09.060549 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:09.060508 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"a41336001b7d0cc13ae18687dfe7caba153e4f2c50f0f538bd78e06e91f57ea7"} Jul 15 19:59:09.062893 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:09.062860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" event={"ID":"c393f5cd-50c2-4e99-a59c-3e4f44a8dd43","Type":"ContainerStarted","Data":"6337b703e710e808b6d1a2d93e8a8c91f319ee376e8d5de7f7d5c80d1e30948e"} Jul 15 19:59:09.070730 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:09.070672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:09.842097 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:09.841869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:09.842292 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:09.842229 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:10.842340 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:10.842092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:10.843077 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:10.842122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:10.843077 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:10.842434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:10.843077 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:10.842476 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:11.069451 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.069416 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="401b8c50370e376a15ede22a2a95801e0459fd2c7e17b4cd456362695d39602d" exitCode=0 Jul 15 19:59:11.069609 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.069506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"401b8c50370e376a15ede22a2a95801e0459fd2c7e17b4cd456362695d39602d"} Jul 15 19:59:11.074670 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.074651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 19:59:11.075058 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.075023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"44bfc5f2cf3577fdd92a03a8c6194456dab08ac26d781a7f2afcace63e677a44"} Jul 15 19:59:11.075348 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.075332 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:11.075414 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.075398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:11.075540 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.075527 2576 scope.go:117] "RemoveContainer" containerID="13bfd0a3638a437b29a8ec1548cb92e7bbb8e189856515b3934659c5e4820caa" Jul 15 19:59:11.091755 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.091731 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:11.101462 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.101412 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9zh72" podStartSLOduration=5.547249969 podStartE2EDuration="28.101396195s" totalImagesPullingTime="22.554146226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:09.099924419 +0000 UTC m=+25.809490672" watchObservedRunningTime="2026-07-15 19:59:11.101396195 +0000 UTC m=+27.810962450" Jul 15 19:59:11.842251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:11.842219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:11.842450 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:11.842379 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:12.081313 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.081290 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 19:59:12.081640 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.081618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" event={"ID":"069a64e5-5074-4274-8b9e-020eeb44415a","Type":"ContainerStarted","Data":"304afd24572868fb9b262885295bc6bce828f5d74c0bc99a8e933b3faf406f6e"} Jul 15 19:59:12.081982 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.081954 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:12.103073 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.102463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:12.114799 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.114739 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" podStartSLOduration=8.866067017 podStartE2EDuration="29.114721053s" totalImagesPullingTime="20.248654036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:12.113442169 +0000 UTC m=+28.823008421" watchObservedRunningTime="2026-07-15 19:59:12.114721053 +0000 UTC m=+28.824287305" Jul 15 19:59:12.506896 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.506809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dfgv2"] Jul 15 19:59:12.507116 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.506995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:12.507157 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:12.507129 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:12.515140 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.515109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-765p2"] Jul 15 19:59:12.515323 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.515228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:12.515370 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:12.515332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:12.524071 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.524034 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vx4g"] Jul 15 19:59:12.524259 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.524164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:12.524303 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:12.524263 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:12.529275 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.529216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:12.628441 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:12.628409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:13.086302 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:13.086262 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="b9637cfea4cab8bc9b674cb4f42cbb63589370ddc95611d82a5445b928caa645" exitCode=0 Jul 15 19:59:13.086832 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:13.086347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"b9637cfea4cab8bc9b674cb4f42cbb63589370ddc95611d82a5445b928caa645"} Jul 15 19:59:13.366268 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:13.366183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:13.422804 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:13.422757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:13.842744 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:13.842716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:13.842907 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:13.842802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:14.598066 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:14.597843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:14.841967 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:14.841932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:14.842165 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:14.842068 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:14.842165 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:14.842128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:14.842286 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:14.842262 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:15.093840 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:15.093752 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="dcdbe7041a5d4a4072d8c1f88ec7154e5976601a24ec3664075dfbd609134520" exitCode=0 Jul 15 19:59:15.093840 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:15.093815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"dcdbe7041a5d4a4072d8c1f88ec7154e5976601a24ec3664075dfbd609134520"} Jul 15 19:59:15.842654 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:15.842613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:15.843365 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:15.842721 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:16.413919 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:16.413884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:16.414140 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:16.413941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:16.414140 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414080 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:59:16.414140 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414093 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:59:16.414300 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414173 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs podName:05151441-6723-414d-85ce-569fa23139d7 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:48.414152236 +0000 UTC m=+65.123718482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs") pod "network-metrics-daemon-2vx4g" (UID: "05151441-6723-414d-85ce-569fa23139d7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:59:16.414300 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414098 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:59:16.414300 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414226 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gm6kd for pod openshift-network-diagnostics/network-check-target-765p2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:59:16.414300 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.414286 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd podName:f5c362b5-a308-48d0-a9c7-9c2a819e4625 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:48.414266108 +0000 UTC m=+65.123832353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gm6kd" (UniqueName: "kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd") pod "network-check-target-765p2" (UID: "f5c362b5-a308-48d0-a9c7-9c2a819e4625") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:59:16.515128 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:16.515093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:16.515314 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.515275 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:59:16.515367 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.515352 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret podName:5ccd8254-4a7f-43c7-aee8-bd36cd411958 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:48.515331521 +0000 UTC m=+65.224897764 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret") pod "global-pull-secret-syncer-dfgv2" (UID: "5ccd8254-4a7f-43c7-aee8-bd36cd411958") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:59:16.842316 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:16.842280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:16.842517 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:16.842299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:16.842517 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.842429 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vx4g" podUID="05151441-6723-414d-85ce-569fa23139d7" Jul 15 19:59:16.842517 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:16.842503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dfgv2" podUID="5ccd8254-4a7f-43c7-aee8-bd36cd411958" Jul 15 19:59:17.842037 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:17.841985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:17.842697 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:17.842129 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-765p2" podUID="f5c362b5-a308-48d0-a9c7-9c2a819e4625" Jul 15 19:59:18.146788 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.146761 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeReady" Jul 15 19:59:18.146991 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.146913 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:59:18.210066 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.210025 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vt4vk"] Jul 15 19:59:18.212480 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.212441 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8b6nk"] Jul 15 19:59:18.212626 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.212593 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.214380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.214356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.218000 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.217973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:59:18.218000 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.217986 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wppjw\"" Jul 15 19:59:18.218212 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.217986 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tvpz5\"" Jul 15 19:59:18.218212 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.217988 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:59:18.218212 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.218003 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:59:18.218212 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.218194 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:59:18.218427 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.218405 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:59:18.226924 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.226871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8b6nk"] Jul 15 19:59:18.235627 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.235589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vt4vk"] Jul 15 19:59:18.329172 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c8a497-4d8b-49a3-843f-16d6445ded03-config-volume\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.329172 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.329430 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0192199e-9b55-451a-8d41-bb9d2a5549bb-cert\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.329430 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7wk\" (UniqueName: \"kubernetes.io/projected/d1c8a497-4d8b-49a3-843f-16d6445ded03-kube-api-access-xz7wk\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.329430 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmhk\" (UniqueName: \"kubernetes.io/projected/0192199e-9b55-451a-8d41-bb9d2a5549bb-kube-api-access-szmhk\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.329625 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.329457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c8a497-4d8b-49a3-843f-16d6445ded03-tmp-dir\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430041 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.429932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7wk\" (UniqueName: \"kubernetes.io/projected/d1c8a497-4d8b-49a3-843f-16d6445ded03-kube-api-access-xz7wk\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430041 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.429980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmhk\" (UniqueName: \"kubernetes.io/projected/0192199e-9b55-451a-8d41-bb9d2a5549bb-kube-api-access-szmhk\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.430272 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c8a497-4d8b-49a3-843f-16d6445ded03-tmp-dir\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430272 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c8a497-4d8b-49a3-843f-16d6445ded03-config-volume\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430272 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430272 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0192199e-9b55-451a-8d41-bb9d2a5549bb-cert\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.430662 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:18.430633 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:59:18.430792 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:18.430711 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls podName:d1c8a497-4d8b-49a3-843f-16d6445ded03 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:18.930690621 +0000 UTC m=+35.640256870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls") pod "dns-default-vt4vk" (UID: "d1c8a497-4d8b-49a3-843f-16d6445ded03") : secret "dns-default-metrics-tls" not found Jul 15 19:59:18.430792 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c8a497-4d8b-49a3-843f-16d6445ded03-tmp-dir\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.430928 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.430837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d1c8a497-4d8b-49a3-843f-16d6445ded03-config-volume\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.435211 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.435183 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0192199e-9b55-451a-8d41-bb9d2a5549bb-cert\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.436651 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.436623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:18.439779 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.439741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7wk\" (UniqueName: \"kubernetes.io/projected/d1c8a497-4d8b-49a3-843f-16d6445ded03-kube-api-access-xz7wk\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.440050 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.440027 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmhk\" (UniqueName: \"kubernetes.io/projected/0192199e-9b55-451a-8d41-bb9d2a5549bb-kube-api-access-szmhk\") pod \"ingress-canary-8b6nk\" (UID: \"0192199e-9b55-451a-8d41-bb9d2a5549bb\") " pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.532183 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.532146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8b6nk" Jul 15 19:59:18.534683 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.534654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:18.841956 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.841918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:18.842139 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.841965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:18.846590 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.846566 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:59:18.846865 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.846841 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:59:18.846865 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.846853 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kl6c8\"" Jul 15 19:59:18.933861 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:18.933825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:18.934129 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:18.934105 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:59:18.934228 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:18.934187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls podName:d1c8a497-4d8b-49a3-843f-16d6445ded03 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:19.93416557 +0000 UTC m=+36.643731816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls") pod "dns-default-vt4vk" (UID: "d1c8a497-4d8b-49a3-843f-16d6445ded03") : secret "dns-default-metrics-tls" not found Jul 15 19:59:19.444937 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.444894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:19.534555 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.534338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:19.636245 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.636203 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8b6nk"] Jul 15 19:59:19.640168 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:19.640132 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0192199e_9b55_451a_8d41_bb9d2a5549bb.slice/crio-b515fe03e75c5d808c1fef74394ed98e87fb0abb48cb3822adc24846d7579828 WatchSource:0}: Error finding container b515fe03e75c5d808c1fef74394ed98e87fb0abb48cb3822adc24846d7579828: Status 404 returned error can't find the container with id b515fe03e75c5d808c1fef74394ed98e87fb0abb48cb3822adc24846d7579828 Jul 15 19:59:19.842262 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.842222 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:19.845225 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.845194 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:59:19.845380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.845249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l6s2m\"" Jul 15 19:59:19.845380 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.845254 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:59:19.942314 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:19.942275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:19.942508 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:19.942426 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:59:19.942508 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:19.942504 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls podName:d1c8a497-4d8b-49a3-843f-16d6445ded03 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:21.942487721 +0000 UTC m=+38.652053951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls") pod "dns-default-vt4vk" (UID: "d1c8a497-4d8b-49a3-843f-16d6445ded03") : secret "dns-default-metrics-tls" not found Jul 15 19:59:20.110192 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.110155 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="6d858d57d03165dfcd48ed2074221092025946bbae48f59bf6cb16010fa2b02e" exitCode=0 Jul 15 19:59:20.110384 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.110234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"6d858d57d03165dfcd48ed2074221092025946bbae48f59bf6cb16010fa2b02e"} Jul 15 19:59:20.111328 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.111306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8b6nk" event={"ID":"0192199e-9b55-451a-8d41-bb9d2a5549bb","Type":"ContainerStarted","Data":"b515fe03e75c5d808c1fef74394ed98e87fb0abb48cb3822adc24846d7579828"} Jul 15 19:59:20.744214 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.744181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:20.843853 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.843802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:20.944155 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.944121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:20.974452 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.974414 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6fbk4"] Jul 15 19:59:20.976847 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.976822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:20.979814 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.979792 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:59:20.980220 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.980157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:59:20.980220 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.980164 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:59:20.981057 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.981037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:59:20.981156 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:20.981037 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v5glv\"" Jul 15 19:59:21.001252 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.001154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fbk4"] Jul 15 19:59:21.036457 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.036420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:21.052276 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.052234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-data-volume\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.052467 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.052306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg6g\" (UniqueName: \"kubernetes.io/projected/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-api-access-dkg6g\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.052467 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.052383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.052467 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.052413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-crio-socket\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.052467 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.052456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.118251 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.118216 2576 generic.go:356] "Generic (PLEG): container finished" podID="79b57609-b496-4140-9565-5b91749c53f0" containerID="e597bc596ee9c7a3623dd3cb0f351d4cf848fd57a1e4f144fd3bd8fbbb9574bc" exitCode=0 Jul 15 19:59:21.118429 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.118268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerDied","Data":"e597bc596ee9c7a3623dd3cb0f351d4cf848fd57a1e4f144fd3bd8fbbb9574bc"} Jul 15 19:59:21.153193 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-crio-socket\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.153382 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.153382 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-data-volume\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.153382 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg6g\" (UniqueName: \"kubernetes.io/projected/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-api-access-dkg6g\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.153544 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.153544 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.153513 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:59:21.153630 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.153585 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:21.653566225 +0000 UTC m=+38.363132469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : configmap "kube-rbac-proxy" not found Jul 15 19:59:21.153691 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.153670 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:21.153753 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.153736 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:21.653715877 +0000 UTC m=+38.363282125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:21.153893 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-crio-socket\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.154000 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.153934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-data-volume\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.178562 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.178527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg6g\" (UniqueName: \"kubernetes.io/projected/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-api-access-dkg6g\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.232585 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.232320 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:21.372954 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.372920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:21.583822 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.583790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:21.658541 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.658447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.658541 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.658485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:21.659045 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.658961 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:59:21.659178 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.659048 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:21.659178 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.659105 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:22.659069996 +0000 UTC m=+39.368636244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : configmap "kube-rbac-proxy" not found Jul 15 19:59:21.659178 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.659148 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:22.659114889 +0000 UTC m=+39.368681328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:21.682043 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.681990 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:21.962417 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:21.962302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:21.963043 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.962452 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:59:21.963043 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:21.962549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls podName:d1c8a497-4d8b-49a3-843f-16d6445ded03 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:25.962521433 +0000 UTC m=+42.672087682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls") pod "dns-default-vt4vk" (UID: "d1c8a497-4d8b-49a3-843f-16d6445ded03") : secret "dns-default-metrics-tls" not found Jul 15 19:59:22.125659 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.125585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" event={"ID":"79b57609-b496-4140-9565-5b91749c53f0","Type":"ContainerStarted","Data":"31d1348b49b00b485583ef4c317bdafe11432d8290354425ee1fb9c48a7d0949"} Jul 15 19:59:22.154287 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.154218 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4s5fc" podStartSLOduration=8.062393331 podStartE2EDuration="39.154196527s" totalImagesPullingTime="31.091803196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:22.152616429 +0000 UTC m=+38.862182689" watchObservedRunningTime="2026-07-15 19:59:22.154196527 +0000 UTC m=+38.863762782" Jul 15 19:59:22.253822 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.253738 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk"] Jul 15 19:59:22.277549 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.277510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk"] Jul 15 19:59:22.277698 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.277591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" Jul 15 19:59:22.281103 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.281076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 19:59:22.281455 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.281432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cjjpf\"" Jul 15 19:59:22.281727 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.281709 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 19:59:22.325306 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.325263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:22.365069 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.365031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6bz\" (UniqueName: \"kubernetes.io/projected/458b315f-8825-4a60-894f-225074e42b5d-kube-api-access-st6bz\") pod \"migrator-7f59b54f6c-pbvvk\" (UID: \"458b315f-8825-4a60-894f-225074e42b5d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" Jul 15 19:59:22.435575 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.435544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:22.466024 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.465972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6bz\" (UniqueName: \"kubernetes.io/projected/458b315f-8825-4a60-894f-225074e42b5d-kube-api-access-st6bz\") pod \"migrator-7f59b54f6c-pbvvk\" (UID: \"458b315f-8825-4a60-894f-225074e42b5d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" Jul 15 19:59:22.484757 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.484717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6bz\" (UniqueName: \"kubernetes.io/projected/458b315f-8825-4a60-894f-225074e42b5d-kube-api-access-st6bz\") pod \"migrator-7f59b54f6c-pbvvk\" (UID: \"458b315f-8825-4a60-894f-225074e42b5d\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" Jul 15 19:59:22.573766 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.573687 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:22.586548 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.586511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" Jul 15 19:59:22.668033 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.667978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:22.668033 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.668040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:22.668222 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:22.668110 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:59:22.668222 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:22.668169 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:22.668222 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:22.668190 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:24.668171609 +0000 UTC m=+41.377737838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : configmap "kube-rbac-proxy" not found Jul 15 19:59:22.668222 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:22.668213 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:24.668202283 +0000 UTC m=+41.377768513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:22.816721 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:22.816684 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk"] Jul 15 19:59:22.820672 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:22.820489 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458b315f_8825_4a60_894f_225074e42b5d.slice/crio-53fcc2dddd41da34fe52f3468be4921827149cc8f0f1555f5fe1f0e4b216d2b1 WatchSource:0}: Error finding container 53fcc2dddd41da34fe52f3468be4921827149cc8f0f1555f5fe1f0e4b216d2b1: Status 404 returned error can't find the container with id 53fcc2dddd41da34fe52f3468be4921827149cc8f0f1555f5fe1f0e4b216d2b1 Jul 15 19:59:23.127164 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.127141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tx46d_8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd/dns-node-resolver/0.log" Jul 15 19:59:23.129346 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.129322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8b6nk" event={"ID":"0192199e-9b55-451a-8d41-bb9d2a5549bb","Type":"ContainerStarted","Data":"fe82f8dc2f1f0492f18d92e382c61d8a402c12d07aa6e27795211ec7cc349cef"} Jul 15 19:59:23.130417 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.130393 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" event={"ID":"458b315f-8825-4a60-894f-225074e42b5d","Type":"ContainerStarted","Data":"53fcc2dddd41da34fe52f3468be4921827149cc8f0f1555f5fe1f0e4b216d2b1"} Jul 15 19:59:23.181026 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.180945 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8b6nk" podStartSLOduration=2.141915844 podStartE2EDuration="5.180923922s" totalImagesPullingTime="3.039008078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:23.159843288 +0000 UTC m=+39.869409537" watchObservedRunningTime="2026-07-15 19:59:23.180923922 +0000 UTC m=+39.890490175" Jul 15 19:59:23.181423 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.181406 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xnmc6"] Jul 15 19:59:23.202772 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.202741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xnmc6"] Jul 15 19:59:23.202952 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.202837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.205888 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.205863 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 19:59:23.206071 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.205862 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 19:59:23.206193 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.206073 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 19:59:23.206266 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.206238 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbbzp\"" Jul 15 19:59:23.206657 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.206632 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 19:59:23.239559 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.239524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:23.365753 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.365718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:23.373803 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.373769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-cabundle\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.373946 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.373863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbt8\" (UniqueName: \"kubernetes.io/projected/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-kube-api-access-sgbt8\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.373946 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.373930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-key\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.474463 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.474384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-key\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.474463 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.474449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-cabundle\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.474626 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.474495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbt8\" (UniqueName: \"kubernetes.io/projected/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-kube-api-access-sgbt8\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.475230 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.475204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-cabundle\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.478150 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.478061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-signing-key\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.487502 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.487469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbt8\" (UniqueName: \"kubernetes.io/projected/c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83-kube-api-access-sgbt8\") pod \"service-ca-79f68b9f4f-xnmc6\" (UID: \"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83\") " pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.514059 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.514002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" Jul 15 19:59:23.732538 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.732411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-xnmc6"] Jul 15 19:59:23.736503 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:23.736446 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6clq5_bf7ab018-02e3-4dd9-bd83-26c6719e02c7/node-ca/0.log" Jul 15 19:59:23.736644 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:23.736606 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc106dc5a_c0eb_4cb8_98f8_1f6dbc63fe83.slice/crio-c25dbcd20afd28ffdac73fd05e6451088baf63e628a67a14816b60f2bf96df20 WatchSource:0}: Error finding container c25dbcd20afd28ffdac73fd05e6451088baf63e628a67a14816b60f2bf96df20: Status 404 returned error can't find the container with id c25dbcd20afd28ffdac73fd05e6451088baf63e628a67a14816b60f2bf96df20 Jul 15 19:59:24.134573 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:24.134525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" event={"ID":"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83","Type":"ContainerStarted","Data":"c25dbcd20afd28ffdac73fd05e6451088baf63e628a67a14816b60f2bf96df20"} Jul 15 19:59:24.683463 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:24.683425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:24.683697 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:24.683472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:24.683697 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:24.683554 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:59:24.683697 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:24.683602 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:24.683697 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:24.683614 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:28.683595514 +0000 UTC m=+45.393161749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : configmap "kube-rbac-proxy" not found Jul 15 19:59:24.683697 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:24.683636 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:28.683626655 +0000 UTC m=+45.393192899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:24.696321 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:24.696284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:24.729643 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:24.729616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8b6nk_0192199e-9b55-451a-8d41-bb9d2a5549bb/serve-healthcheck-canary/0.log" Jul 15 19:59:25.139921 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.139874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" event={"ID":"458b315f-8825-4a60-894f-225074e42b5d","Type":"ContainerStarted","Data":"535ae509dbfab5cce7e98f361cf6a72ffdd2d8cac5f437589f54b1800499f5cb"} Jul 15 19:59:25.140386 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.139949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" event={"ID":"458b315f-8825-4a60-894f-225074e42b5d","Type":"ContainerStarted","Data":"85e19d374d67fd1ab78abaa80fe79cd2b1f7ff54786851a650ecf35e9b1d5464"} Jul 15 19:59:25.168696 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.168636 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-pbvvk" podStartSLOduration=1.422425023 podStartE2EDuration="3.168609035s" totalImagesPullingTime="1.746184012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:25.16677146 +0000 UTC m=+41.876337712" watchObservedRunningTime="2026-07-15 19:59:25.168609035 +0000 UTC m=+41.878175285" Jul 15 19:59:25.570960 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.570879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:25.671335 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.671282 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:25.995612 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:25.995567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:25.995801 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:25.995777 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:59:25.995888 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:25.995866 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls podName:d1c8a497-4d8b-49a3-843f-16d6445ded03 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:33.995842629 +0000 UTC m=+50.705408896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls") pod "dns-default-vt4vk" (UID: "d1c8a497-4d8b-49a3-843f-16d6445ded03") : secret "dns-default-metrics-tls" not found Jul 15 19:59:27.146225 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:27.146183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" event={"ID":"c106dc5a-c0eb-4cb8-98f8-1f6dbc63fe83","Type":"ContainerStarted","Data":"6be8fc526927943407026657cc2d84df78e04a02a336d352655a9ea982b62d1c"} Jul 15 19:59:27.166635 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:27.166583 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-xnmc6" podStartSLOduration=1.4070758620000001 podStartE2EDuration="4.166567126s" totalImagesPullingTime="2.759491264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:27.165318149 +0000 UTC m=+43.874884403" watchObservedRunningTime="2026-07-15 19:59:27.166567126 +0000 UTC m=+43.876133377" Jul 15 19:59:27.741145 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:27.741063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:27.966636 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:27.966599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:28.094429 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.094385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:28.213226 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.213159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:28.309526 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.309476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:28.422791 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.422694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:28.717796 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.717697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:28.717796 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:28.717754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:28.718116 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:28.717819 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:59:28.718116 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:28.717890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:36.71787115 +0000 UTC m=+53.427437398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : configmap "kube-rbac-proxy" not found Jul 15 19:59:28.718116 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:28.717937 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:28.718116 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:28.718021 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:36.717986971 +0000 UTC m=+53.427553260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:29.021714 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:29.021632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:29.443135 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:29.443105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:30.220158 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:30.220116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:30.315331 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:30.315300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:30.619350 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:30.619317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:30.821145 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:30.821109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:31.016037 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:31.015929 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:31.415707 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:31.415671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:31.516144 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:31.516113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:31.949581 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:31.949548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:32.051333 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:32.051296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:32.225449 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:32.225359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:32.656239 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:32.656197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:32.753612 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:32.753577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:33.234791 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:33.234753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:33.628472 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:33.628438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:34.037381 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.037288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:34.062420 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.062382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:34.065739 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.065701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d1c8a497-4d8b-49a3-843f-16d6445ded03-metrics-tls\") pod \"dns-default-vt4vk\" (UID: \"d1c8a497-4d8b-49a3-843f-16d6445ded03\") " pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:34.124977 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.124935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:34.137330 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.137298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:34.237963 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.237852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:34.273031 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.272979 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vt4vk"] Jul 15 19:59:34.276221 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:34.276185 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c8a497_4d8b_49a3_843f_16d6445ded03.slice/crio-2c08b401af8066207fac4de4b8a1f391d46ba817168a0d00d089a0ce8b1d9cb4 WatchSource:0}: Error finding container 2c08b401af8066207fac4de4b8a1f391d46ba817168a0d00d089a0ce8b1d9cb4: Status 404 returned error can't find the container with id 2c08b401af8066207fac4de4b8a1f391d46ba817168a0d00d089a0ce8b1d9cb4 Jul 15 19:59:34.337139 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.337103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:34.703878 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:34.703792 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:35.037454 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:35.037360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:35.171732 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:35.171693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vt4vk" event={"ID":"d1c8a497-4d8b-49a3-843f-16d6445ded03","Type":"ContainerStarted","Data":"2c08b401af8066207fac4de4b8a1f391d46ba817168a0d00d089a0ce8b1d9cb4"} Jul 15 19:59:35.612410 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:35.612380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:35.859894 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:35.859821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:36.226548 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.226520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:36.415118 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.415082 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:36.632444 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.632399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:36.781164 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.781114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:36.781353 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.781174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:36.781418 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:36.781367 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:59:36.781469 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:36.781422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls podName:fcf59bdb-94a6-4b31-a825-9c7c1bdb163f nodeName:}" failed. No retries permitted until 2026-07-15 19:59:52.781406843 +0000 UTC m=+69.490973073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6fbk4" (UID: "fcf59bdb-94a6-4b31-a825-9c7c1bdb163f") : secret "insights-runtime-extractor-tls" not found Jul 15 19:59:36.782142 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.782113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:36.991565 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:36.991525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:37.097880 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.097772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:37.179499 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.179457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vt4vk" event={"ID":"d1c8a497-4d8b-49a3-843f-16d6445ded03","Type":"ContainerStarted","Data":"b7275b5f1fb8b133c525d581c38c4a38e394f63cf0334ff33bc875b555ae5781"} Jul 15 19:59:37.179673 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.179546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vt4vk" event={"ID":"d1c8a497-4d8b-49a3-843f-16d6445ded03","Type":"ContainerStarted","Data":"f4b079b86d624cf4b64def74e02a6e5078608279b0d859e3ac8d2ee1223715b5"} Jul 15 19:59:37.179673 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.179585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:37.209999 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.209946 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vt4vk" podStartSLOduration=16.896845561 podStartE2EDuration="19.20992861s" totalImagesPullingTime="2.313083049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:37.209212698 +0000 UTC m=+53.918778950" watchObservedRunningTime="2026-07-15 19:59:37.20992861 +0000 UTC m=+53.919494861" Jul 15 19:59:37.475174 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:37.475093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:38.060070 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:38.060029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:38.884346 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:38.884315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.007533 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.007484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.100087 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.100058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.198935 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.198843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.349028 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.348971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.583505 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.583474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.767973 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.767936 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:39.872242 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:39.872158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:40.088222 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:40.088169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:40.229692 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:40.229606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:40.752795 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:40.752762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:40.946555 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:40.946518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:41.025309 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:41.025218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:42.223621 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:42.223577 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:42.342114 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:42.342081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:43.373620 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:43.373588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:43.439608 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:43.439570 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:44.108451 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:44.108419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dkmdg" Jul 15 19:59:46.787629 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:46.787595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:47.007770 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.007735 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:47.184871 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.184835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vt4vk" Jul 15 19:59:47.185971 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.185949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:47.672641 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.672610 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr"] Jul 15 19:59:47.675501 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.675480 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj"] Jul 15 19:59:47.675621 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.675604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.677332 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.677305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.702436 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.702391 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.703297 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.703258 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" Jul 15 19:59:47.709920 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.709897 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" Jul 15 19:59:47.710184 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.710161 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" Jul 15 19:59:47.714179 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.714156 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 15 19:59:47.739527 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.739492 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" Jul 15 19:59:47.739991 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.739972 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" Jul 15 19:59:47.740568 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.740543 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" Jul 15 19:59:47.740648 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.740570 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 15 19:59:47.745188 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.745145 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="ceedfa93-cd2d-481f-ae92-6eaa630d8c01" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.745296 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:47.745195 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-f2mbx\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f2mbx\"" type="*v1.Secret" Jul 15 19:59:47.766778 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.766778 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.767002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.767002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.767002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.767002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ccg\" (UniqueName: \"kubernetes.io/projected/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-kube-api-access-r4ccg\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.767002 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.766978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9sb\" (UniqueName: \"kubernetes.io/projected/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-kube-api-access-qp9sb\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.767231 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.767063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.782125 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.782092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr"] Jul 15 19:59:47.782774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.782753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj"] Jul 15 19:59:47.868339 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.868339 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ccg\" (UniqueName: \"kubernetes.io/projected/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-kube-api-access-r4ccg\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9sb\" (UniqueName: \"kubernetes.io/projected/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-kube-api-access-qp9sb\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:47.868834 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:47.868606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:48.212414 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.212384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:48.406204 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.406167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:48.474266 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.474166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:48.474266 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.474236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:48.478671 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.478643 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:59:48.478789 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.478728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:59:48.487373 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.487348 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05151441-6723-414d-85ce-569fa23139d7-metrics-certs\") pod \"network-metrics-daemon-2vx4g\" (UID: \"05151441-6723-414d-85ce-569fa23139d7\") " pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:48.488806 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.488786 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:59:48.498401 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.498375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6kd\" (UniqueName: \"kubernetes.io/projected/f5c362b5-a308-48d0-a9c7-9c2a819e4625-kube-api-access-gm6kd\") pod \"network-check-target-765p2\" (UID: \"f5c362b5-a308-48d0-a9c7-9c2a819e4625\") " pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:48.535067 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.535037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 19:59:48.539785 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.539758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:48.551992 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.551965 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 19:59:48.559294 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.559267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kl6c8\"" Jul 15 19:59:48.566264 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.566236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vx4g" Jul 15 19:59:48.574790 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.574697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:48.577774 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.577749 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 19:59:48.578530 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.578510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:59:48.586654 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.586581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9sb\" (UniqueName: \"kubernetes.io/projected/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-kube-api-access-qp9sb\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:48.587470 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.587445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ccd8254-4a7f-43c7-aee8-bd36cd411958-original-pull-secret\") pod \"global-pull-secret-syncer-dfgv2\" (UID: \"5ccd8254-4a7f-43c7-aee8-bd36cd411958\") " pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:48.588752 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.588728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ccg\" (UniqueName: \"kubernetes.io/projected/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-kube-api-access-r4ccg\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:48.618652 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.618624 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f2mbx\"" Jul 15 19:59:48.663502 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.663459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l6s2m\"" Jul 15 19:59:48.673389 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.673177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:48.740293 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.740217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vx4g"] Jul 15 19:59:48.743662 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:48.743632 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05151441_6723_414d_85ce_569fa23139d7.slice/crio-6b3eb501f28b65a9e339e41eba08bf8202f74c805f47837f630a10b900adfcdb WatchSource:0}: Error finding container 6b3eb501f28b65a9e339e41eba08bf8202f74c805f47837f630a10b900adfcdb: Status 404 returned error can't find the container with id 6b3eb501f28b65a9e339e41eba08bf8202f74c805f47837f630a10b900adfcdb Jul 15 19:59:48.810724 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.810690 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-765p2"] Jul 15 19:59:48.813566 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:48.813539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c362b5_a308_48d0_a9c7_9c2a819e4625.slice/crio-daa3422076b45bf3dbbd1a30b98f7625585b1f45057561a95bc19185421f2260 WatchSource:0}: Error finding container daa3422076b45bf3dbbd1a30b98f7625585b1f45057561a95bc19185421f2260: Status 404 returned error can't find the container with id daa3422076b45bf3dbbd1a30b98f7625585b1f45057561a95bc19185421f2260 Jul 15 19:59:48.861860 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.861813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dfgv2" Jul 15 19:59:48.866253 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.866232 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 19:59:48.869252 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869230 2576 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869258 2576 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869230 2576 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert: failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869309 2576 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/managed-serviceaccount-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869317 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig podName:a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc nodeName:}" failed. No retries permitted until 2026-07-15 19:59:49.369285716 +0000 UTC m=+66.078851961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig") pod "cluster-proxy-proxy-agent-55d7fbb666-cffsr" (UID: "a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc") : failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca podName:a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc nodeName:}" failed. No retries permitted until 2026-07-15 19:59:49.369358533 +0000 UTC m=+66.078924766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca") pod "cluster-proxy-proxy-agent-55d7fbb666-cffsr" (UID: "a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc") : failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869395 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub podName:a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc nodeName:}" failed. No retries permitted until 2026-07-15 19:59:49.369386476 +0000 UTC m=+66.078952712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub" (UniqueName: "kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub") pod "cluster-proxy-proxy-agent-55d7fbb666-cffsr" (UID: "a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc") : failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.869613 ip-10-0-141-87 kubenswrapper[2576]: E0715 19:59:48.869410 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig podName:ceedfa93-cd2d-481f-ae92-6eaa630d8c01 nodeName:}" failed. No retries permitted until 2026-07-15 19:59:49.36940154 +0000 UTC m=+66.078967784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig") pod "managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" (UID: "ceedfa93-cd2d-481f-ae92-6eaa630d8c01") : failed to sync secret cache: timed out waiting for the condition Jul 15 19:59:48.871570 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.871551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:48.913400 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.913362 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 19:59:48.926806 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:48.926775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:49.008260 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.008239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dfgv2"] Jul 15 19:59:49.010057 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:49.010030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccd8254_4a7f_43c7_aee8_bd36cd411958.slice/crio-bdf07397261546e995a8b6696888753eabd5ca3bb1bd1612f4e038e10be75878 WatchSource:0}: Error finding container bdf07397261546e995a8b6696888753eabd5ca3bb1bd1612f4e038e10be75878: Status 404 returned error can't find the container with id bdf07397261546e995a8b6696888753eabd5ca3bb1bd1612f4e038e10be75878 Jul 15 19:59:49.111574 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.111543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 19:59:49.215973 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.215930 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-765p2" event={"ID":"f5c362b5-a308-48d0-a9c7-9c2a819e4625","Type":"ContainerStarted","Data":"daa3422076b45bf3dbbd1a30b98f7625585b1f45057561a95bc19185421f2260"} Jul 15 19:59:49.217033 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.216988 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dfgv2" event={"ID":"5ccd8254-4a7f-43c7-aee8-bd36cd411958","Type":"ContainerStarted","Data":"bdf07397261546e995a8b6696888753eabd5ca3bb1bd1612f4e038e10be75878"} Jul 15 19:59:49.218086 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.218063 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vx4g" event={"ID":"05151441-6723-414d-85ce-569fa23139d7","Type":"ContainerStarted","Data":"6b3eb501f28b65a9e339e41eba08bf8202f74c805f47837f630a10b900adfcdb"} Jul 15 19:59:49.281539 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.281457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 19:59:49.288963 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.288937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 19:59:49.381939 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.381901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:49.382151 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.381947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.382151 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.381971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.382151 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.382005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.384939 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.384905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-ca\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.385268 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.385243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.385517 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.385489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceedfa93-cd2d-481f-ae92-6eaa630d8c01-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj\" (UID: \"ceedfa93-cd2d-481f-ae92-6eaa630d8c01\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:49.386884 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.386864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc-hub\") pod \"cluster-proxy-proxy-agent-55d7fbb666-cffsr\" (UID: \"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.501122 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.501028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" Jul 15 19:59:49.505310 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.505225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" Jul 15 19:59:49.941483 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.941452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj"] Jul 15 19:59:49.946704 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:49.946662 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceedfa93_cd2d_481f_ae92_6eaa630d8c01.slice/crio-50568a5a61ce732a42e3d32d680a0e610419d26a015669ca20ee1827f0e711a5 WatchSource:0}: Error finding container 50568a5a61ce732a42e3d32d680a0e610419d26a015669ca20ee1827f0e711a5: Status 404 returned error can't find the container with id 50568a5a61ce732a42e3d32d680a0e610419d26a015669ca20ee1827f0e711a5 Jul 15 19:59:49.965749 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:49.965717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr"] Jul 15 19:59:50.225168 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.224630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vx4g" event={"ID":"05151441-6723-414d-85ce-569fa23139d7","Type":"ContainerStarted","Data":"f9bdcf5611f9749f03f10f430e8a07c2c2d2dc67f933fe0678ba86f4ffa115f1"} Jul 15 19:59:50.225168 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.224691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vx4g" event={"ID":"05151441-6723-414d-85ce-569fa23139d7","Type":"ContainerStarted","Data":"500829544f75c4864caf3c8e829073de89ad21f6d64d841d92bce350a42a6d4d"} Jul 15 19:59:50.226171 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.226133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" event={"ID":"ceedfa93-cd2d-481f-ae92-6eaa630d8c01","Type":"ContainerStarted","Data":"50568a5a61ce732a42e3d32d680a0e610419d26a015669ca20ee1827f0e711a5"} Jul 15 19:59:50.227700 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.227674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" event={"ID":"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc","Type":"ContainerStarted","Data":"99bfb3e48d4f95b1ebaef1c98957abfd4446ac62187f2132e04a69a4d8db305e"} Jul 15 19:59:50.402511 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.402469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:50.507361 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:50.507272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:51.000163 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:51.000088 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:51.104539 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:51.104499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:52.813652 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:52.813595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:52.816146 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:52.816121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcf59bdb-94a6-4b31-a825-9c7c1bdb163f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6fbk4\" (UID: \"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f\") " pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:53.091784 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:53.091755 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v5glv\"" Jul 15 19:59:53.099605 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:53.099523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6fbk4" Jul 15 19:59:56.523925 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:56.523726 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2vx4g" podStartSLOduration=72.465151933 podStartE2EDuration="1m13.523707725s" totalImagesPullingTime="1.058555792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:50.274428613 +0000 UTC m=+66.983994869" watchObservedRunningTime="2026-07-15 19:59:56.523707725 +0000 UTC m=+73.233273976" Jul 15 19:59:56.523925 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:56.523920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6fbk4"] Jul 15 19:59:56.530097 ip-10-0-141-87 kubenswrapper[2576]: W0715 19:59:56.529878 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf59bdb_94a6_4b31_a825_9c7c1bdb163f.slice/crio-6c19ffc0cb6e235c34cb7033a64e5abf80488388cdda7f863fe096a295b70ee2 WatchSource:0}: Error finding container 6c19ffc0cb6e235c34cb7033a64e5abf80488388cdda7f863fe096a295b70ee2: Status 404 returned error can't find the container with id 6c19ffc0cb6e235c34cb7033a64e5abf80488388cdda7f863fe096a295b70ee2 Jul 15 19:59:56.734096 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:56.734058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:56.837809 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:56.837770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:56.921497 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:56.921413 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:57.260950 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.260846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-765p2" event={"ID":"f5c362b5-a308-48d0-a9c7-9c2a819e4625","Type":"ContainerStarted","Data":"912d2c2df65b843dba27ba563df81d5ddbc993a9fd81472c16827da55d63bf41"} Jul 15 19:59:57.261686 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.261611 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 19:59:57.264182 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.264003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dfgv2" event={"ID":"5ccd8254-4a7f-43c7-aee8-bd36cd411958","Type":"ContainerStarted","Data":"e0238b32a18395f9c360ab2752788f50ee08f6ee36eebea17fca8bcaf730bb09"} Jul 15 19:59:57.266252 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.266216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" event={"ID":"ceedfa93-cd2d-481f-ae92-6eaa630d8c01","Type":"ContainerStarted","Data":"0d61daf392cb7836a3eaace8ae7d0e3e27fc1938d404898ff821c74c85ecae1a"} Jul 15 19:59:57.271980 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.271456 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" event={"ID":"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc","Type":"ContainerStarted","Data":"5df908efafe64b76ed59b3cf20bc822a9307abd7acadcecfcd98d5466e71f83d"} Jul 15 19:59:57.271980 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.271499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" event={"ID":"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc","Type":"ContainerStarted","Data":"25b1a359d63ae917ea731e9c268a4fbc09618f0fde987919baea30054953d9bc"} Jul 15 19:59:57.271980 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.271516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" event={"ID":"a24772ec-9d80-48a5-85a3-b0d1b0bfb6cc","Type":"ContainerStarted","Data":"52693444cfd3d293f4a7d02b08b18b9410f2007ccbb45743f56d9b3267a875c9"} Jul 15 19:59:57.273680 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.273610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fbk4" event={"ID":"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f","Type":"ContainerStarted","Data":"6c19ffc0cb6e235c34cb7033a64e5abf80488388cdda7f863fe096a295b70ee2"} Jul 15 19:59:57.300609 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.299723 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-765p2" podStartSLOduration=66.74481024 podStartE2EDuration="1m14.299702286s" totalImagesPullingTime="7.554892046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:57.298361291 +0000 UTC m=+74.007927544" watchObservedRunningTime="2026-07-15 19:59:57.299702286 +0000 UTC m=+74.009268542" Jul 15 19:59:57.355937 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.354794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7549f6ccf4-wk6vj" podStartSLOduration=3.924398556 podStartE2EDuration="10.354772185s" totalImagesPullingTime="6.430373629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:57.330832665 +0000 UTC m=+74.040398947" watchObservedRunningTime="2026-07-15 19:59:57.354772185 +0000 UTC m=+74.064338440" Jul 15 19:59:57.355937 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.355198 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dfgv2" podStartSLOduration=66.99646308 podStartE2EDuration="1m14.355185711s" totalImagesPullingTime="7.358722631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:57.354169907 +0000 UTC m=+74.063736161" watchObservedRunningTime="2026-07-15 19:59:57.355185711 +0000 UTC m=+74.064751964" Jul 15 19:59:57.375764 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.374780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d7fbb666-cffsr" podStartSLOduration=3.962046531 podStartE2EDuration="10.374759754s" totalImagesPullingTime="6.412713223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:57.373296096 +0000 UTC m=+74.082862351" watchObservedRunningTime="2026-07-15 19:59:57.374759754 +0000 UTC m=+74.084325988" Jul 15 19:59:57.575915 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.575823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:57.648591 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:57.648552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:58.034060 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:58.034022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:58.112686 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:58.112637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:58.946919 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:58.946884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:59.765331 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:59.765295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:59.866582 ip-10-0-141-87 kubenswrapper[2576]: I0715 19:59:59.866547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:01.223510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:01.223470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:01.312962 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:01.312930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:01.647388 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:01.647356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:01.752229 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:01.752197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:02.292992 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:02.292959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fbk4" event={"ID":"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f","Type":"ContainerStarted","Data":"d7de569e2dcf043b861f7eef9c26e3d0d4324e666b789ce207faf8f533ec16cc"} Jul 15 20:00:02.615601 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:02.615567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:02.712125 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:02.712093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:06.310254 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:06.310219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fbk4" event={"ID":"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f","Type":"ContainerStarted","Data":"bb50adde40ad8005e12151b5f8713a650c6600ac29605b0db947c4dd9fa689bf"} Jul 15 20:00:06.789972 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:06.789940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:07.113140 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:07.113108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:07.436176 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:07.436094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:07.533835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:07.533798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:10.473142 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:10.473108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:10.601874 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:10.601841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:11.331181 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:11.331147 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6fbk4" event={"ID":"fcf59bdb-94a6-4b31-a825-9c7c1bdb163f","Type":"ContainerStarted","Data":"001642ae1b6dab5bd4f596320e201b88ff69493a241ac62842c5b5dada973048"} Jul 15 20:00:11.360823 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:11.360765 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6fbk4" podStartSLOduration=37.479631012 podStartE2EDuration="51.360748744s" totalImagesPullingTime="13.881117732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:59:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:00:11.359912037 +0000 UTC m=+88.069478288" watchObservedRunningTime="2026-07-15 20:00:11.360748744 +0000 UTC m=+88.070314995" Jul 15 20:00:12.570615 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:12.570575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:12.630958 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:12.630926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:12.705447 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:12.705417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:12.807072 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:12.807039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:13.366894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:13.366852 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:13.423237 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:13.423201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:14.747283 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:14.747238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:14.847086 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:14.847046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:15.057531 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:15.057443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:16.148302 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:16.148267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:16.249701 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:16.249656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:17.318748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.318718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:17.690994 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.690956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:17.780368 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.780337 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9qwnw"] Jul 15 20:00:17.783819 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.783801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.786371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.786346 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f5fs6\"" Jul 15 20:00:17.787438 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787408 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 20:00:17.787557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 20:00:17.787557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787463 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 20:00:17.787732 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 20:00:17.787785 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787715 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 20:00:17.787862 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.787846 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 20:00:17.812883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.812847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:17.897129 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-metrics-client-ca\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-wtmp\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-sys\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-root\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-tls\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcr77\" (UniqueName: \"kubernetes.io/projected/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-kube-api-access-mcr77\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.897523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.897411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-textfile\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.975057 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.974956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:17.998120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-metrics-client-ca\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-wtmp\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-sys\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-root\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-tls\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-sys\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-root\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998360 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-wtmp\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcr77\" (UniqueName: \"kubernetes.io/projected/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-kube-api-access-mcr77\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-textfile\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998842 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-metrics-client-ca\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.998954 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-textfile\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:17.999031 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:17.998935 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:18.001088 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.001068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-tls\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:18.001185 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.001169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:18.014267 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.014235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcr77\" (UniqueName: \"kubernetes.io/projected/07c11d81-3bf5-4022-9a88-e4a2a3de4a7f-kube-api-access-mcr77\") pod \"node-exporter-9qwnw\" (UID: \"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f\") " pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:18.095530 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.095495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9qwnw" Jul 15 20:00:18.108235 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:00:18.108209 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c11d81_3bf5_4022_9a88_e4a2a3de4a7f.slice/crio-cbb75abeb26c618eef3bf8cf710a4754b19eafa48e19082973e65d75457ecd95 WatchSource:0}: Error finding container cbb75abeb26c618eef3bf8cf710a4754b19eafa48e19082973e65d75457ecd95: Status 404 returned error can't find the container with id cbb75abeb26c618eef3bf8cf710a4754b19eafa48e19082973e65d75457ecd95 Jul 15 20:00:18.124548 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.124518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:18.314446 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.314362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:18.354966 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.354934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qwnw" event={"ID":"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f","Type":"ContainerStarted","Data":"cbb75abeb26c618eef3bf8cf710a4754b19eafa48e19082973e65d75457ecd95"} Jul 15 20:00:18.446783 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.446750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:18.716341 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.716298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:18.842030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:18.841977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:19.025446 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:19.025327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:19.360116 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:19.360076 2576 generic.go:356] "Generic (PLEG): container finished" podID="07c11d81-3bf5-4022-9a88-e4a2a3de4a7f" containerID="ae44deca2bba609f16b1a8a08df7a43046ab3c2a42002bb371b69b15513a8403" exitCode=0 Jul 15 20:00:19.360463 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:19.360135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qwnw" event={"ID":"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f","Type":"ContainerDied","Data":"ae44deca2bba609f16b1a8a08df7a43046ab3c2a42002bb371b69b15513a8403"} Jul 15 20:00:19.654294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:19.654241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:19.732607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:19.732565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:20.364825 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:20.364796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qwnw" event={"ID":"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f","Type":"ContainerStarted","Data":"95a712c6bc495f7f3f3988d9048abaae57445b3fb3755e1710bcf09828f94d9d"} Jul 15 20:00:20.365251 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:20.364846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qwnw" event={"ID":"07c11d81-3bf5-4022-9a88-e4a2a3de4a7f","Type":"ContainerStarted","Data":"01596bfcbe4e8ec9d229e9d49cac4542396e08b93305cd5e26b2d0467a95c3f1"} Jul 15 20:00:20.408057 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:20.407990 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9qwnw" podStartSLOduration=2.310004885 podStartE2EDuration="3.407974657s" totalImagesPullingTime="1.031554568s" totalInitContainerRuntime="66.415204ms" isStatefulPod=true podCreationTimestamp="2026-07-15 20:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:00:20.407691295 +0000 UTC m=+97.117257547" watchObservedRunningTime="2026-07-15 20:00:20.407974657 +0000 UTC m=+97.117540941" Jul 15 20:00:20.763146 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:20.763054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:20.823256 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:20.823219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:21.109189 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:21.109160 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:21.210271 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:21.210237 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.236750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:23.236718 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.311100 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:23.311061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.536395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:23.536319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.670560 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:23.670527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.769999 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:23.769961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:24.072537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:24.072474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:28.394339 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:28.394299 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:28.505246 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:28.505210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:29.283619 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:29.283587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-765p2" Jul 15 20:00:39.004083 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:39.004041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:39.153905 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:39.153866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:40.067481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:40.067446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:40.177213 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:40.177178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:43.392842 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:43.392805 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:43.508186 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:00:43.508146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:33.994655 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:33.994596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:34.181797 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:34.181760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:39.201078 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:39.200977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:39.335396 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:39.335359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:40.095612 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:40.095583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:46.021421 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:46.021384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:46.113380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:46.113336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:50.452679 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:50.452648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:50.552754 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:50.552721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:50.621670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:50.621633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:59.236331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:59.236289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:59.337634 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:01:59.337598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:03.066611 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:03.066578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:03.162915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:03.162877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:24.833685 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:24.833649 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:55.154959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:55.154917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:55.279481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:55.279446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:58.387281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:58.387221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:58.463946 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:02:58.463915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:05.899149 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:03:05.899081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:05.968322 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:03:05.968289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:43.666425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:03:43.666392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:03:43.668943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:03:43.668921 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 20:03:43.673436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:03:43.673411 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:04:28.222053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:28.221979 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:31.537713 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:31.537683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:32.814830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:32.814776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:34.027325 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:34.027290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:34.521233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:34.521194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:34.694306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:34.694268 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:34.897363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:34.897330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:36.997499 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:36.997464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:37.452149 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:37.452111 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:42.887924 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.887886 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6cbvv"] Jul 15 20:04:42.889925 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.889907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:42.892653 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.892630 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 20:04:42.893838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.893821 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7m88x\"" Jul 15 20:04:42.893918 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.893848 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 20:04:42.900475 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.900450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6cbvv"] Jul 15 20:04:42.941030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.940981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z52j9\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-kube-api-access-z52j9\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:42.941214 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:42.941113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.041407 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.041368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.041567 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.041413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z52j9\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-kube-api-access-z52j9\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.052201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.052173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.052337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.052207 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z52j9\" (UniqueName: \"kubernetes.io/projected/11755db3-ff5e-458b-b20f-ec0c1bfeb475-kube-api-access-z52j9\") pod \"cert-manager-webhook-6486ffd997-6cbvv\" (UID: \"11755db3-ff5e-458b-b20f-ec0c1bfeb475\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.198877 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.198785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:43.338926 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.338892 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6cbvv"] Jul 15 20:04:43.342544 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:04:43.342515 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11755db3_ff5e_458b_b20f_ec0c1bfeb475.slice/crio-4550808aa5fe29ea0aea48b4aed785997904c20153533e42ef6459a65cc3b1c4 WatchSource:0}: Error finding container 4550808aa5fe29ea0aea48b4aed785997904c20153533e42ef6459a65cc3b1c4: Status 404 returned error can't find the container with id 4550808aa5fe29ea0aea48b4aed785997904c20153533e42ef6459a65cc3b1c4 Jul 15 20:04:43.344459 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:43.344438 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:04:44.232590 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:44.232551 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" event={"ID":"11755db3-ff5e-458b-b20f-ec0c1bfeb475","Type":"ContainerStarted","Data":"4550808aa5fe29ea0aea48b4aed785997904c20153533e42ef6459a65cc3b1c4"} Jul 15 20:04:44.982990 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:44.982940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:45.404247 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:45.404212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:45.566429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:45.566392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:46.609294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.609265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:46.751264 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.751227 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s"] Jul 15 20:04:46.753537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.753518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.756529 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.756502 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 20:04:46.756881 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.756865 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 20:04:46.758933 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.758911 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2mvmp\"" Jul 15 20:04:46.764265 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.764241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:46.767846 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.767819 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s"] Jul 15 20:04:46.875485 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.875383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n4h\" (UniqueName: \"kubernetes.io/projected/6b67276a-a46a-470e-b64c-1d262b58dd62-kube-api-access-25n4h\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.875485 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.875460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b67276a-a46a-470e-b64c-1d262b58dd62-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.976625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.976587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n4h\" (UniqueName: \"kubernetes.io/projected/6b67276a-a46a-470e-b64c-1d262b58dd62-kube-api-access-25n4h\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.976625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.976627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b67276a-a46a-470e-b64c-1d262b58dd62-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.976987 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.976968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b67276a-a46a-470e-b64c-1d262b58dd62-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:46.992607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:46.992577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n4h\" (UniqueName: \"kubernetes.io/projected/6b67276a-a46a-470e-b64c-1d262b58dd62-kube-api-access-25n4h\") pod \"openshift-lws-operator-fd8ccff4c-s6n9s\" (UID: \"6b67276a-a46a-470e-b64c-1d262b58dd62\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:47.062749 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:47.062701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" Jul 15 20:04:47.245695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:47.245654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" event={"ID":"11755db3-ff5e-458b-b20f-ec0c1bfeb475","Type":"ContainerStarted","Data":"9239795c51e1fda9bb6c9f403c6b255c43c82ca50ed495bfa9b06807dfa0a5f4"} Jul 15 20:04:47.245899 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:47.245794 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:04:47.266383 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:04:47.266350 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b67276a_a46a_470e_b64c_1d262b58dd62.slice/crio-c96c275fadd235e33a1e8cd1fb1e0de28d4f139b5cf6ef55c21395ba214afeaa WatchSource:0}: Error finding container c96c275fadd235e33a1e8cd1fb1e0de28d4f139b5cf6ef55c21395ba214afeaa: Status 404 returned error can't find the container with id c96c275fadd235e33a1e8cd1fb1e0de28d4f139b5cf6ef55c21395ba214afeaa Jul 15 20:04:47.277633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:47.277600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s"] Jul 15 20:04:47.301288 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:47.301233 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" podStartSLOduration=2.321324007 podStartE2EDuration="5.30121809s" totalImagesPullingTime="2.979894083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:04:47.30096303 +0000 UTC m=+364.010529283" watchObservedRunningTime="2026-07-15 20:04:47.30121809 +0000 UTC m=+364.010784343" Jul 15 20:04:48.252021 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:48.251974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" event={"ID":"6b67276a-a46a-470e-b64c-1d262b58dd62","Type":"ContainerStarted","Data":"c96c275fadd235e33a1e8cd1fb1e0de28d4f139b5cf6ef55c21395ba214afeaa"} Jul 15 20:04:50.260863 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:50.260826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" event={"ID":"6b67276a-a46a-470e-b64c-1d262b58dd62","Type":"ContainerStarted","Data":"495ec08a3f9195579f5d8de8904b30b09a38a635cce9bb51a95191ecd61110ad"} Jul 15 20:04:50.291754 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:50.291698 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s6n9s" podStartSLOduration=1.774233779 podStartE2EDuration="4.291681589s" totalImagesPullingTime="2.51744781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:04:50.289742645 +0000 UTC m=+366.999308896" watchObservedRunningTime="2026-07-15 20:04:50.291681589 +0000 UTC m=+367.001247840" Jul 15 20:04:51.208470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:51.208434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:51.370488 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:51.370447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:53.027373 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:53.027338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:04:53.254177 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:04:53.254143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6cbvv" Jul 15 20:05:02.156146 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:02.156108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:05.362611 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:05.362574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:05.725415 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:05.725333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:05.916229 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:05.916190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:07.164671 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.164642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:07.340186 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.340157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:07.444070 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.443988 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8"] Jul 15 20:05:07.447051 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.447033 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.458724 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.458705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 20:05:07.459130 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.459114 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b8zwh\"" Jul 15 20:05:07.459851 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.459832 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 20:05:07.460198 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.460184 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 20:05:07.468962 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.468938 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 20:05:07.482952 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.482923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8"] Jul 15 20:05:07.537775 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.537733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hlx\" (UniqueName: \"kubernetes.io/projected/1c8ba5a7-509f-445f-a8d7-0d46952bc305-kube-api-access-j4hlx\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.537951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.537791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.537951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.537813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-manifests\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.537951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.537829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.537951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.537869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-tmp\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638499 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638499 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-manifests\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-tmp\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hlx\" (UniqueName: \"kubernetes.io/projected/1c8ba5a7-509f-445f-a8d7-0d46952bc305-kube-api-access-j4hlx\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.638986 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.638916 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-manifests\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.640934 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.640913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8ba5a7-509f-445f-a8d7-0d46952bc305-tmp\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.641142 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.641109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.641188 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.641140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8ba5a7-509f-445f-a8d7-0d46952bc305-webhook-cert\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.675417 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.675385 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hlx\" (UniqueName: \"kubernetes.io/projected/1c8ba5a7-509f-445f-a8d7-0d46952bc305-kube-api-access-j4hlx\") pod \"opendatahub-operator-controller-manager-56f6d85d69-9qhw8\" (UID: \"1c8ba5a7-509f-445f-a8d7-0d46952bc305\") " pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.757309 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.757213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:07.924317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:07.924290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8"] Jul 15 20:05:07.925401 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:07.925368 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8ba5a7_509f_445f_a8d7_0d46952bc305.slice/crio-9c594e61c9a438a017f54a6643aa972b68bcbfe387c6a6f44ea4ad514a886f79 WatchSource:0}: Error finding container 9c594e61c9a438a017f54a6643aa972b68bcbfe387c6a6f44ea4ad514a886f79: Status 404 returned error can't find the container with id 9c594e61c9a438a017f54a6643aa972b68bcbfe387c6a6f44ea4ad514a886f79 Jul 15 20:05:08.063265 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.061390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:08.213189 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.213149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:08.323999 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.323910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" event={"ID":"1c8ba5a7-509f-445f-a8d7-0d46952bc305","Type":"ContainerStarted","Data":"9c594e61c9a438a017f54a6643aa972b68bcbfe387c6a6f44ea4ad514a886f79"} Jul 15 20:05:08.363575 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.363542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:08.497161 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.497130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:08.667981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.667939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:08.829069 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:08.828999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:09.552284 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:09.552217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:09.872166 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:09.872131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:11.338917 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:11.338884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" event={"ID":"1c8ba5a7-509f-445f-a8d7-0d46952bc305","Type":"ContainerStarted","Data":"733a08d538bb634c8589438f5efdd968c246cc5cd153082e61977d55fa4d3e34"} Jul 15 20:05:12.344306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:12.344267 2576 generic.go:356] "Generic (PLEG): container finished" podID="1c8ba5a7-509f-445f-a8d7-0d46952bc305" containerID="733a08d538bb634c8589438f5efdd968c246cc5cd153082e61977d55fa4d3e34" exitCode=0 Jul 15 20:05:12.344694 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:12.344357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" event={"ID":"1c8ba5a7-509f-445f-a8d7-0d46952bc305","Type":"ContainerDied","Data":"733a08d538bb634c8589438f5efdd968c246cc5cd153082e61977d55fa4d3e34"} Jul 15 20:05:15.356897 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:15.356861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" event={"ID":"1c8ba5a7-509f-445f-a8d7-0d46952bc305","Type":"ContainerStarted","Data":"10d99f317774f075bd0a9ec17724f80c4823d4d73f03139b67d39c7acd8f4263"} Jul 15 20:05:15.357314 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:15.357001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:15.403275 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:15.403225 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" podStartSLOduration=1.595253441 podStartE2EDuration="8.40320956s" totalImagesPullingTime="5.710029568s" totalInitContainerRuntime="1.097926551s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:15.39840542 +0000 UTC m=+392.107971676" watchObservedRunningTime="2026-07-15 20:05:15.40320956 +0000 UTC m=+392.112775811" Jul 15 20:05:26.363334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:26.363296 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56f6d85d69-9qhw8" Jul 15 20:05:29.216365 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:29.216330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:29.640437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:29.640400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:29.954670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:29.954589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:30.198965 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.198925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:30.228730 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.228631 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9"] Jul 15 20:05:30.237935 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.237907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.262729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.262697 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 20:05:30.262915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.262901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 20:05:30.263816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.263795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 20:05:30.291150 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.291119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 20:05:30.307513 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.307480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dzc\" (UniqueName: \"kubernetes.io/projected/86dddc3b-e6d9-45ba-9014-3233d427e6c3-kube-api-access-d7dzc\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.307657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.307528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tmp\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.307657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.307549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tls-certs\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.319872 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.319834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9"] Jul 15 20:05:30.408792 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.408745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tmp\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.408792 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.408798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tls-certs\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.409103 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.408856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dzc\" (UniqueName: \"kubernetes.io/projected/86dddc3b-e6d9-45ba-9014-3233d427e6c3-kube-api-access-d7dzc\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.411153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.411122 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tmp\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.411341 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.411325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86dddc3b-e6d9-45ba-9014-3233d427e6c3-tls-certs\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.429158 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.429123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dzc\" (UniqueName: \"kubernetes.io/projected/86dddc3b-e6d9-45ba-9014-3233d427e6c3-kube-api-access-d7dzc\") pod \"kube-auth-proxy-6dcf7fc97d-r4hd9\" (UID: \"86dddc3b-e6d9-45ba-9014-3233d427e6c3\") " pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.456138 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.456102 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2xmqq"] Jul 15 20:05:30.458496 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.458476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.462450 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.462432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cqjhb\"" Jul 15 20:05:30.462542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.462524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 20:05:30.489635 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.489538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2xmqq"] Jul 15 20:05:30.509434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.509395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdsf\" (UniqueName: \"kubernetes.io/projected/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-kube-api-access-bvdsf\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.509434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.509452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.545805 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.545762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:30.547671 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.547650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" Jul 15 20:05:30.610986 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.610945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdsf\" (UniqueName: \"kubernetes.io/projected/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-kube-api-access-bvdsf\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.611226 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.611039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.611352 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:05:30.611241 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 20:05:30.611352 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:05:30.611307 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert podName:aa76a2a8-8617-4d30-930d-44b2d8f4b9a8 nodeName:}" failed. No retries permitted until 2026-07-15 20:05:31.111284753 +0000 UTC m=+407.820850995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert") pod "odh-model-controller-7998c67dc4-2xmqq" (UID: "aa76a2a8-8617-4d30-930d-44b2d8f4b9a8") : secret "odh-model-controller-webhook-cert" not found Jul 15 20:05:30.625978 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.625926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdsf\" (UniqueName: \"kubernetes.io/projected/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-kube-api-access-bvdsf\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:30.731644 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.731619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9"] Jul 15 20:05:30.734075 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:30.734041 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dddc3b_e6d9_45ba_9014_3233d427e6c3.slice/crio-2d6cb9a072b2e83ebe29eaed142e21d83ef059cc7fea35547495a58343768358 WatchSource:0}: Error finding container 2d6cb9a072b2e83ebe29eaed142e21d83ef059cc7fea35547495a58343768358: Status 404 returned error can't find the container with id 2d6cb9a072b2e83ebe29eaed142e21d83ef059cc7fea35547495a58343768358 Jul 15 20:05:30.834139 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:30.834105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:31.115848 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.115733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:31.118303 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.118278 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa76a2a8-8617-4d30-930d-44b2d8f4b9a8-cert\") pod \"odh-model-controller-7998c67dc4-2xmqq\" (UID: \"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8\") " pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:31.128300 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.128271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:31.306352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.306314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:31.368828 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.368724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:31.423763 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.423647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" event={"ID":"86dddc3b-e6d9-45ba-9014-3233d427e6c3","Type":"ContainerStarted","Data":"2d6cb9a072b2e83ebe29eaed142e21d83ef059cc7fea35547495a58343768358"} Jul 15 20:05:31.543569 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:31.543383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2xmqq"] Jul 15 20:05:31.546164 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:31.546137 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa76a2a8_8617_4d30_930d_44b2d8f4b9a8.slice/crio-a6a1bafdcc5bacc395f0f16b840234716794704a07e2a47e5af6285de54c833d WatchSource:0}: Error finding container a6a1bafdcc5bacc395f0f16b840234716794704a07e2a47e5af6285de54c833d: Status 404 returned error can't find the container with id a6a1bafdcc5bacc395f0f16b840234716794704a07e2a47e5af6285de54c833d Jul 15 20:05:32.104505 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.104471 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ht6rv"] Jul 15 20:05:32.106630 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.106604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.112583 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.112561 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 20:05:32.124346 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.124307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjld\" (UniqueName: \"kubernetes.io/projected/e7a239d1-3a23-44fc-95c5-384d5d7490a1-kube-api-access-jjjld\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.124534 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.124447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.204059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.203878 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ht6rv"] Jul 15 20:05:32.225949 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.225905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjld\" (UniqueName: \"kubernetes.io/projected/e7a239d1-3a23-44fc-95c5-384d5d7490a1-kube-api-access-jjjld\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.226161 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.225987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.226233 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:05:32.226190 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 15 20:05:32.226286 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:05:32.226277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs podName:e7a239d1-3a23-44fc-95c5-384d5d7490a1 nodeName:}" failed. No retries permitted until 2026-07-15 20:05:32.726253889 +0000 UTC m=+409.435820142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs") pod "model-serving-api-54d89bd5fc-ht6rv" (UID: "e7a239d1-3a23-44fc-95c5-384d5d7490a1") : secret "model-serving-api-tls" not found Jul 15 20:05:32.228152 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.228120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:32.244579 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.244537 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjld\" (UniqueName: \"kubernetes.io/projected/e7a239d1-3a23-44fc-95c5-384d5d7490a1-kube-api-access-jjjld\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.404382 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.404297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:32.430782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.430746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" event={"ID":"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8","Type":"ContainerStarted","Data":"a6a1bafdcc5bacc395f0f16b840234716794704a07e2a47e5af6285de54c833d"} Jul 15 20:05:32.729355 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.729263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.732309 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.732271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a239d1-3a23-44fc-95c5-384d5d7490a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-ht6rv\" (UID: \"e7a239d1-3a23-44fc-95c5-384d5d7490a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:32.897945 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.897902 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:32.900660 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.900630 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.903944 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.903912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 20:05:32.904218 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.904198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 20:05:32.905313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.904406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 20:05:32.905313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.904648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hqn7q\"" Jul 15 20:05:32.905313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.904851 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 20:05:32.923278 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.923243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:32.925754 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.925724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:32.932030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.931979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbt5\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:32.932444 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:32.932217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.014345 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.014258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:33.019239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.019207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbt5\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.033942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.033838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.034611 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.034503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.037392 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.037219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.037392 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.037309 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.037392 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.037392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.037608 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.037558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.062077 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.062041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.065463 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.065393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbt5\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5\") pod \"istiod-openshift-gateway-7574459656-qcz5d\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.211061 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.210911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ht6rv"] Jul 15 20:05:33.215645 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:33.215607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:33.864735 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:33.864695 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a239d1_3a23_44fc_95c5_384d5d7490a1.slice/crio-33c7f6c5fa1324279b25a68b2e6b5202417900cc3b92d7dbae222b17737441f9 WatchSource:0}: Error finding container 33c7f6c5fa1324279b25a68b2e6b5202417900cc3b92d7dbae222b17737441f9: Status 404 returned error can't find the container with id 33c7f6c5fa1324279b25a68b2e6b5202417900cc3b92d7dbae222b17737441f9 Jul 15 20:05:34.395380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:34.395343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:34.441342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:34.441305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" event={"ID":"e7a239d1-3a23-44fc-95c5-384d5d7490a1","Type":"ContainerStarted","Data":"33c7f6c5fa1324279b25a68b2e6b5202417900cc3b92d7dbae222b17737441f9"} Jul 15 20:05:35.201806 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.201768 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:35.205757 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:35.205723 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0a7ecb_2b75_4db9_b890_4e383696fe90.slice/crio-3d37ec337c40f3b5e327439880995de84ea00698275d5f8f5ce45ad3c4fb4b70 WatchSource:0}: Error finding container 3d37ec337c40f3b5e327439880995de84ea00698275d5f8f5ce45ad3c4fb4b70: Status 404 returned error can't find the container with id 3d37ec337c40f3b5e327439880995de84ea00698275d5f8f5ce45ad3c4fb4b70 Jul 15 20:05:35.447960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.447920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" event={"ID":"aa76a2a8-8617-4d30-930d-44b2d8f4b9a8","Type":"ContainerStarted","Data":"02a18d6515cac68566d9d9da8c7661deb287739e994d1b5acf072957779b9c57"} Jul 15 20:05:35.448165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.447989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:35.449885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.449850 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" event={"ID":"86dddc3b-e6d9-45ba-9014-3233d427e6c3","Type":"ContainerStarted","Data":"4e7fe8f7f5d76420e93801818becddafd01f2b3dca1c4e80a2831ac343a8e9e1"} Jul 15 20:05:35.451246 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.451218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" event={"ID":"6a0a7ecb-2b75-4db9-b890-4e383696fe90","Type":"ContainerStarted","Data":"3d37ec337c40f3b5e327439880995de84ea00698275d5f8f5ce45ad3c4fb4b70"} Jul 15 20:05:35.469907 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.469840 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" podStartSLOduration=1.9546249260000002 podStartE2EDuration="5.469816259s" totalImagesPullingTime="3.515191333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:35.468944857 +0000 UTC m=+412.178511108" watchObservedRunningTime="2026-07-15 20:05:35.469816259 +0000 UTC m=+412.179382512" Jul 15 20:05:35.502574 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:35.502506 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dcf7fc97d-r4hd9" podStartSLOduration=1.180798741 podStartE2EDuration="5.502486769s" totalImagesPullingTime="4.321688028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:35.498601801 +0000 UTC m=+412.208168070" watchObservedRunningTime="2026-07-15 20:05:35.502486769 +0000 UTC m=+412.212053151" Jul 15 20:05:37.465317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:37.465152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" event={"ID":"e7a239d1-3a23-44fc-95c5-384d5d7490a1","Type":"ContainerStarted","Data":"700f41d96aef6f91828b710cfec06c5bd8137f84ee0022817221253e078d7ef4"} Jul 15 20:05:37.465786 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:37.465346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:37.505364 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:37.505306 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" podStartSLOduration=2.618306651 podStartE2EDuration="5.50528047s" totalImagesPullingTime="2.886973819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:37.503472601 +0000 UTC m=+414.213038862" watchObservedRunningTime="2026-07-15 20:05:37.50528047 +0000 UTC m=+414.214846722" Jul 15 20:05:37.868428 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:37.868385 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:05:37.868529 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:37.868466 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:05:38.220231 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.220136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:38.319852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.319821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:38.471282 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.471182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" event={"ID":"6a0a7ecb-2b75-4db9-b890-4e383696fe90","Type":"ContainerStarted","Data":"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e"} Jul 15 20:05:38.471721 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.471429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:38.473421 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.473378 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-qcz5d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:05:38.473570 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.473438 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:05:38.501461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.501394 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" podStartSLOduration=3.841311365 podStartE2EDuration="6.501374199s" totalImagesPullingTime="2.660062834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:38.498902685 +0000 UTC m=+415.208468938" watchObservedRunningTime="2026-07-15 20:05:38.501374199 +0000 UTC m=+415.210940454" Jul 15 20:05:38.661317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.661265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:38.785625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:38.785536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:39.181608 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.181572 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq"] Jul 15 20:05:39.185392 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.185368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.191947 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.191914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xwc25\"" Jul 15 20:05:39.253444 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.253403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:39.290245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290353 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnp5\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-kube-api-access-gjnp5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.290650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.290544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.297684 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.297648 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq"] Jul 15 20:05:39.391141 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnp5\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-kube-api-access-gjnp5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391606 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.391906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.391843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.392173 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.392152 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.393499 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.393476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.393904 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.393883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.408027 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.407986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.413964 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.413922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnp5\" (UniqueName: \"kubernetes.io/projected/d9eca3c8-6b0c-4314-b73e-6cd47143443f-kube-api-access-gjnp5\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4fpxhq\" (UID: \"d9eca3c8-6b0c-4314-b73e-6cd47143443f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.475993 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.475904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:39.496232 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.496196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:39.537312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.537274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:39.658688 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.658653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:39.659379 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:39.659358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq"] Jul 15 20:05:39.662282 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:39.662256 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9eca3c8_6b0c_4314_b73e_6cd47143443f.slice/crio-8fe05db323536623b43368f04223081ebd8841766d9fcc000e2d8b19b8e6092b WatchSource:0}: Error finding container 8fe05db323536623b43368f04223081ebd8841766d9fcc000e2d8b19b8e6092b: Status 404 returned error can't find the container with id 8fe05db323536623b43368f04223081ebd8841766d9fcc000e2d8b19b8e6092b Jul 15 20:05:40.481491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:40.481459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" event={"ID":"d9eca3c8-6b0c-4314-b73e-6cd47143443f","Type":"ContainerStarted","Data":"8fe05db323536623b43368f04223081ebd8841766d9fcc000e2d8b19b8e6092b"} Jul 15 20:05:42.154941 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.154898 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:05:42.155224 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.154979 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:05:42.155224 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.155026 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:05:42.458422 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.458332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:42.458579 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.458509 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" containerID="cri-o://1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e" gracePeriod=30 Jul 15 20:05:42.476213 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.476171 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-qcz5d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.19:8080/ready\": dial tcp 10.134.0.19:8080: connect: connection refused" start-of-body= Jul 15 20:05:42.476357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.476230 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.19:8080/ready\": dial tcp 10.134.0.19:8080: connect: connection refused" Jul 15 20:05:42.495273 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.495226 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" event={"ID":"d9eca3c8-6b0c-4314-b73e-6cd47143443f","Type":"ContainerStarted","Data":"2577d1c0470580c7f6e40e86ba90bf63285f661278667a0be7514371f1a4d4c5"} Jul 15 20:05:42.496510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.496484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:42.567454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.567400 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" podStartSLOduration=1.077103649 podStartE2EDuration="3.567378595s" totalImagesPullingTime="2.490274946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:42.566465218 +0000 UTC m=+419.276031471" watchObservedRunningTime="2026-07-15 20:05:42.567378595 +0000 UTC m=+419.276944913" Jul 15 20:05:42.708102 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.708068 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:42.824932 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.824850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbt5\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.824932 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.824913 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.824953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.824985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.825040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.825075 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.825103 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts\") pod \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\" (UID: \"6a0a7ecb-2b75-4db9-b890-4e383696fe90\") " Jul 15 20:05:42.825500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.825470 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:05:42.827817 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.827769 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:05:42.827950 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.827810 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:05:42.827950 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.827811 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts" (OuterVolumeSpecName: "cacerts") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:05:42.827950 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.827818 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs" (OuterVolumeSpecName: "local-certs") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:05:42.828140 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.827950 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:05:42.828140 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.828001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5" (OuterVolumeSpecName: "kube-api-access-lzbt5") pod "6a0a7ecb-2b75-4db9-b890-4e383696fe90" (UID: "6a0a7ecb-2b75-4db9-b890-4e383696fe90"). InnerVolumeSpecName "kube-api-access-lzbt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:05:42.926204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926164 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-kubeconfig\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926200 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926213 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-ca-configmap\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926222 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-istio-csr-dns-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926231 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a0a7ecb-2b75-4db9-b890-4e383696fe90-cacerts\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926241 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbt5\" (UniqueName: \"kubernetes.io/projected/6a0a7ecb-2b75-4db9-b890-4e383696fe90-kube-api-access-lzbt5\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:42.926453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:42.926250 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a0a7ecb-2b75-4db9-b890-4e383696fe90-local-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:05:43.205904 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.205863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:43.392782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.392739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n"] Jul 15 20:05:43.393074 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.393059 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" Jul 15 20:05:43.393074 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.393074 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" Jul 15 20:05:43.393202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.393130 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerName="discovery" Jul 15 20:05:43.467984 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.467897 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm"] Jul 15 20:05:43.468185 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.468064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.490351 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.490315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n"] Jul 15 20:05:43.490519 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.490429 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.496238 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.496202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 20:05:43.496568 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.496541 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z762x\"" Jul 15 20:05:43.501743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.501713 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:43.502425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502398 2576 generic.go:356] "Generic (PLEG): container finished" podID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" containerID="1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e" exitCode=0 Jul 15 20:05:43.502523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502444 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" Jul 15 20:05:43.502523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" event={"ID":"6a0a7ecb-2b75-4db9-b890-4e383696fe90","Type":"ContainerDied","Data":"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e"} Jul 15 20:05:43.502590 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d" event={"ID":"6a0a7ecb-2b75-4db9-b890-4e383696fe90","Type":"ContainerDied","Data":"3d37ec337c40f3b5e327439880995de84ea00698275d5f8f5ce45ad3c4fb4b70"} Jul 15 20:05:43.502590 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502544 2576 scope.go:117] "RemoveContainer" containerID="1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e" Jul 15 20:05:43.502953 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.502926 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:43.503983 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.503721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4fpxhq" Jul 15 20:05:43.505456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.505434 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:43.511850 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.511826 2576 scope.go:117] "RemoveContainer" containerID="1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e" Jul 15 20:05:43.512266 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:05:43.512229 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e\": container with ID starting with 1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e not found: ID does not exist" containerID="1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e" Jul 15 20:05:43.512348 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.512271 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e"} err="failed to get container status \"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e\": rpc error: code = NotFound desc = could not find container \"1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e\": container with ID starting with 1c4387a6771b821f3a36df7388e894d6cedd03562ed6e167d147606b6653dd3e not found: ID does not exist" Jul 15 20:05:43.514205 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.514182 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 20:05:43.514845 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.514827 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vqjw2\"" Jul 15 20:05:43.524649 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.524616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm"] Jul 15 20:05:43.530797 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.530767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dtx\" (UniqueName: \"kubernetes.io/projected/66131016-f44c-4f4b-bc91-bdc733e13185-kube-api-access-p2dtx\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.530797 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.530801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.530977 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.530821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjpf\" (UniqueName: \"kubernetes.io/projected/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-kube-api-access-kzjpf\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.530977 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.530929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66131016-f44c-4f4b-bc91-bdc733e13185-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.605334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.605283 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:43.618092 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.618054 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-qcz5d"] Jul 15 20:05:43.632296 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.632266 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjpf\" (UniqueName: \"kubernetes.io/projected/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-kube-api-access-kzjpf\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.632411 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.632352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66131016-f44c-4f4b-bc91-bdc733e13185-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.632457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.632422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dtx\" (UniqueName: \"kubernetes.io/projected/66131016-f44c-4f4b-bc91-bdc733e13185-kube-api-access-p2dtx\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.632457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.632448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.634956 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.634922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66131016-f44c-4f4b-bc91-bdc733e13185-cert\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.634956 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.634945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.692185 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.692153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjpf\" (UniqueName: \"kubernetes.io/projected/2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2-kube-api-access-kzjpf\") pod \"llmisvc-controller-manager-777c6bd7d5-8m5rm\" (UID: \"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.713408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.713370 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dtx\" (UniqueName: \"kubernetes.io/projected/66131016-f44c-4f4b-bc91-bdc733e13185-kube-api-access-p2dtx\") pod \"kserve-controller-manager-d6f4b65b8-zxm8n\" (UID: \"66131016-f44c-4f4b-bc91-bdc733e13185\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.757035 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.756930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:43.785859 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.785820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z762x\"" Jul 15 20:05:43.788866 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.788838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:43.805492 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.805459 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vqjw2\"" Jul 15 20:05:43.811584 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.811560 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:43.850679 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:43.850642 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0a7ecb-2b75-4db9-b890-4e383696fe90" path="/var/lib/kubelet/pods/6a0a7ecb-2b75-4db9-b890-4e383696fe90/volumes" Jul 15 20:05:44.016943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.016850 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n"] Jul 15 20:05:44.020623 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:44.020595 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66131016_f44c_4f4b_bc91_bdc733e13185.slice/crio-651cda367207e711f3f58b4f63d5171dc450119ecf74e0d43541afdc1e5fe52f WatchSource:0}: Error finding container 651cda367207e711f3f58b4f63d5171dc450119ecf74e0d43541afdc1e5fe52f: Status 404 returned error can't find the container with id 651cda367207e711f3f58b4f63d5171dc450119ecf74e0d43541afdc1e5fe52f Jul 15 20:05:44.038205 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.038172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:44.044786 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.044757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm"] Jul 15 20:05:44.047548 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:05:44.047522 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2af7ed50_bdbe_4fef_bbcc_1df601c9c9c2.slice/crio-6312a3e0c7b2dad869a7b8909f40d09ef7016317772495434612166a45d9bc8b WatchSource:0}: Error finding container 6312a3e0c7b2dad869a7b8909f40d09ef7016317772495434612166a45d9bc8b: Status 404 returned error can't find the container with id 6312a3e0c7b2dad869a7b8909f40d09ef7016317772495434612166a45d9bc8b Jul 15 20:05:44.160759 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.160714 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:44.508758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.508718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" event={"ID":"66131016-f44c-4f4b-bc91-bdc733e13185","Type":"ContainerStarted","Data":"651cda367207e711f3f58b4f63d5171dc450119ecf74e0d43541afdc1e5fe52f"} Jul 15 20:05:44.509924 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:44.509899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" event={"ID":"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2","Type":"ContainerStarted","Data":"6312a3e0c7b2dad869a7b8909f40d09ef7016317772495434612166a45d9bc8b"} Jul 15 20:05:46.460241 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:46.460212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2xmqq" Jul 15 20:05:48.475676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:48.475635 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ht6rv" Jul 15 20:05:48.536566 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:48.536528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" event={"ID":"2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2","Type":"ContainerStarted","Data":"253ab5a7ed68d15c6a580c1a1fe362181693ec4e884c8be20f5a9f752b976f57"} Jul 15 20:05:48.536735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:48.536669 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:05:48.645179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:48.645119 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" podStartSLOduration=2.20687843 podStartE2EDuration="5.64510113s" totalImagesPullingTime="3.4382227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:48.643178131 +0000 UTC m=+425.352744398" watchObservedRunningTime="2026-07-15 20:05:48.64510113 +0000 UTC m=+425.354667407" Jul 15 20:05:49.542600 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:49.542565 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" event={"ID":"66131016-f44c-4f4b-bc91-bdc733e13185","Type":"ContainerStarted","Data":"5368f9d3ba57c0a4f678c26e3df8adf4bd9b736ac4751e7830009917edabc241"} Jul 15 20:05:49.571849 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:49.571798 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" podStartSLOduration=2.080553176 podStartE2EDuration="6.571778812s" totalImagesPullingTime="4.491225636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:05:49.570723083 +0000 UTC m=+426.280289336" watchObservedRunningTime="2026-07-15 20:05:49.571778812 +0000 UTC m=+426.281345064" Jul 15 20:05:49.693062 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:49.693027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:49.845359 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:49.845324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:05:50.546912 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:50.546879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:05:51.409400 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:05:51.409369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:19.545470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:19.545389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8m5rm" Jul 15 20:06:21.556540 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:21.556508 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zxm8n" Jul 15 20:06:30.456929 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.456892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:30.575097 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.575066 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:30.585615 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.585588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:30.588915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.588882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 20:06:30.589117 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.588884 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 20:06:30.589771 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.589746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7v2fd\"" Jul 15 20:06:30.591237 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.591206 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:30.594588 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.594505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:30.621187 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.621147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5m7\" (UniqueName: \"kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7\") pod \"kuadrant-operator-catalog-25b4r\" (UID: \"30629ec0-6412-4415-b1eb-1e22d1cc6039\") " pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:30.685506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.685472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:30.722668 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.722574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5m7\" (UniqueName: \"kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7\") pod \"kuadrant-operator-catalog-25b4r\" (UID: \"30629ec0-6412-4415-b1eb-1e22d1cc6039\") " pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:30.731822 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.731790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5m7\" (UniqueName: \"kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7\") pod \"kuadrant-operator-catalog-25b4r\" (UID: \"30629ec0-6412-4415-b1eb-1e22d1cc6039\") " pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:30.825800 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.825763 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:30.826069 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.826040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:30.965840 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:30.965804 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:30.969581 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:06:30.969553 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30629ec0_6412_4415_b1eb_1e22d1cc6039.slice/crio-2aa8d6d2334de70c34d0257be481245b6f9e73990b59a08ca8ec3ebd3a1c48c4 WatchSource:0}: Error finding container 2aa8d6d2334de70c34d0257be481245b6f9e73990b59a08ca8ec3ebd3a1c48c4: Status 404 returned error can't find the container with id 2aa8d6d2334de70c34d0257be481245b6f9e73990b59a08ca8ec3ebd3a1c48c4 Jul 15 20:06:31.044921 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.044888 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cslc6"] Jul 15 20:06:31.048744 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.048571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:31.067444 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.067412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cslc6"] Jul 15 20:06:31.126514 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.126472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp6kd\" (UniqueName: \"kubernetes.io/projected/afb2e1b7-f79f-4d44-879f-fc3d4a563b75-kube-api-access-dp6kd\") pod \"kuadrant-operator-catalog-cslc6\" (UID: \"afb2e1b7-f79f-4d44-879f-fc3d4a563b75\") " pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:31.226985 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.226881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp6kd\" (UniqueName: \"kubernetes.io/projected/afb2e1b7-f79f-4d44-879f-fc3d4a563b75-kube-api-access-dp6kd\") pod \"kuadrant-operator-catalog-cslc6\" (UID: \"afb2e1b7-f79f-4d44-879f-fc3d4a563b75\") " pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:31.237988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.237950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp6kd\" (UniqueName: \"kubernetes.io/projected/afb2e1b7-f79f-4d44-879f-fc3d4a563b75-kube-api-access-dp6kd\") pod \"kuadrant-operator-catalog-cslc6\" (UID: \"afb2e1b7-f79f-4d44-879f-fc3d4a563b75\") " pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:31.271461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.271430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:31.360468 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.360426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:31.474522 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.474419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:06:31.509293 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.509204 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cslc6"] Jul 15 20:06:31.511920 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:06:31.511888 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb2e1b7_f79f_4d44_879f_fc3d4a563b75.slice/crio-71052a0a8fdb55cdff91e5e6b28c1e70312bb6adb362922167ca51702b204d1a WatchSource:0}: Error finding container 71052a0a8fdb55cdff91e5e6b28c1e70312bb6adb362922167ca51702b204d1a: Status 404 returned error can't find the container with id 71052a0a8fdb55cdff91e5e6b28c1e70312bb6adb362922167ca51702b204d1a Jul 15 20:06:31.719484 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.719452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" event={"ID":"30629ec0-6412-4415-b1eb-1e22d1cc6039","Type":"ContainerStarted","Data":"2aa8d6d2334de70c34d0257be481245b6f9e73990b59a08ca8ec3ebd3a1c48c4"} Jul 15 20:06:31.720530 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:31.720505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" event={"ID":"afb2e1b7-f79f-4d44-879f-fc3d4a563b75","Type":"ContainerStarted","Data":"71052a0a8fdb55cdff91e5e6b28c1e70312bb6adb362922167ca51702b204d1a"} Jul 15 20:06:33.730532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.730492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" event={"ID":"30629ec0-6412-4415-b1eb-1e22d1cc6039","Type":"ContainerStarted","Data":"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737"} Jul 15 20:06:33.730532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.730512 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" podUID="30629ec0-6412-4415-b1eb-1e22d1cc6039" containerName="registry-server" containerID="cri-o://a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737" gracePeriod=2 Jul 15 20:06:33.732004 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.731980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" event={"ID":"afb2e1b7-f79f-4d44-879f-fc3d4a563b75","Type":"ContainerStarted","Data":"9ae98d885a5e6d953eb5fc8f4cbebe303cf4cf39af6eecf9ad8c64fe2f1379fd"} Jul 15 20:06:33.748673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.748592 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" podStartSLOduration=1.601689002 podStartE2EDuration="3.748573448s" totalImagesPullingTime="2.146884446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:06:33.747107681 +0000 UTC m=+470.456673935" watchObservedRunningTime="2026-07-15 20:06:33.748573448 +0000 UTC m=+470.458139700" Jul 15 20:06:33.765671 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.765608 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" podStartSLOduration=1.157510845 podStartE2EDuration="2.76559118s" totalImagesPullingTime="1.608080335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:06:33.764707851 +0000 UTC m=+470.474274104" watchObservedRunningTime="2026-07-15 20:06:33.76559118 +0000 UTC m=+470.475157432" Jul 15 20:06:33.982730 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:33.982660 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:34.051446 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.051413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5m7\" (UniqueName: \"kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7\") pod \"30629ec0-6412-4415-b1eb-1e22d1cc6039\" (UID: \"30629ec0-6412-4415-b1eb-1e22d1cc6039\") " Jul 15 20:06:34.053641 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.053614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7" (OuterVolumeSpecName: "kube-api-access-ww5m7") pod "30629ec0-6412-4415-b1eb-1e22d1cc6039" (UID: "30629ec0-6412-4415-b1eb-1e22d1cc6039"). InnerVolumeSpecName "kube-api-access-ww5m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:06:34.152176 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.152122 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5m7\" (UniqueName: \"kubernetes.io/projected/30629ec0-6412-4415-b1eb-1e22d1cc6039-kube-api-access-ww5m7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:06:34.737091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.737050 2576 generic.go:356] "Generic (PLEG): container finished" podID="30629ec0-6412-4415-b1eb-1e22d1cc6039" containerID="a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737" exitCode=0 Jul 15 20:06:34.737091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.737096 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" Jul 15 20:06:34.737564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.737129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" event={"ID":"30629ec0-6412-4415-b1eb-1e22d1cc6039","Type":"ContainerDied","Data":"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737"} Jul 15 20:06:34.737564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.737178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-25b4r" event={"ID":"30629ec0-6412-4415-b1eb-1e22d1cc6039","Type":"ContainerDied","Data":"2aa8d6d2334de70c34d0257be481245b6f9e73990b59a08ca8ec3ebd3a1c48c4"} Jul 15 20:06:34.737564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.737199 2576 scope.go:117] "RemoveContainer" containerID="a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737" Jul 15 20:06:34.746762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.746738 2576 scope.go:117] "RemoveContainer" containerID="a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737" Jul 15 20:06:34.747063 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:06:34.747040 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737\": container with ID starting with a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737 not found: ID does not exist" containerID="a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737" Jul 15 20:06:34.747134 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.747076 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737"} err="failed to get container status \"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737\": rpc error: code = NotFound desc = could not find container \"a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737\": container with ID starting with a90d76eb000f5036d6ca30f7a7cd298ad264264c8367290d3bb094b9e56e7737 not found: ID does not exist" Jul 15 20:06:34.773798 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.771097 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:34.775717 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:34.775684 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-25b4r"] Jul 15 20:06:35.847764 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:35.847723 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30629ec0-6412-4415-b1eb-1e22d1cc6039" path="/var/lib/kubelet/pods/30629ec0-6412-4415-b1eb-1e22d1cc6039/volumes" Jul 15 20:06:41.361477 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:41.361436 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:41.361904 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:41.361496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:41.384979 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:41.384947 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:06:41.789512 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:06:41.789436 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cslc6" Jul 15 20:07:01.241735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:01.241700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:03.941561 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.941520 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc"] Jul 15 20:07:03.942168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.941974 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30629ec0-6412-4415-b1eb-1e22d1cc6039" containerName="registry-server" Jul 15 20:07:03.942168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.941989 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30629ec0-6412-4415-b1eb-1e22d1cc6039" containerName="registry-server" Jul 15 20:07:03.942168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.942083 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30629ec0-6412-4415-b1eb-1e22d1cc6039" containerName="registry-server" Jul 15 20:07:03.946617 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.946590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:03.949998 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:03.949969 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tj5zk\"" Jul 15 20:07:04.023709 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.023668 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc"] Jul 15 20:07:04.099852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.099811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrpv\" (UniqueName: \"kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv\") pod \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" (UID: \"a75c5e33-236f-4726-87d4-f3edf06d437b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:04.201310 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.201214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrpv\" (UniqueName: \"kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv\") pod \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" (UID: \"a75c5e33-236f-4726-87d4-f3edf06d437b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:04.218725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.218693 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrpv\" (UniqueName: \"kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv\") pod \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" (UID: \"a75c5e33-236f-4726-87d4-f3edf06d437b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:04.256688 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.256649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:04.402106 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.402070 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc"] Jul 15 20:07:04.404387 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:04.404362 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75c5e33_236f_4726_87d4_f3edf06d437b.slice/crio-06807aaca02285c050d3dc086c73f6d19f13aed14346e26491de0a7d9c5a091c WatchSource:0}: Error finding container 06807aaca02285c050d3dc086c73f6d19f13aed14346e26491de0a7d9c5a091c: Status 404 returned error can't find the container with id 06807aaca02285c050d3dc086c73f6d19f13aed14346e26491de0a7d9c5a091c Jul 15 20:07:04.867551 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:04.867506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" event={"ID":"a75c5e33-236f-4726-87d4-f3edf06d437b","Type":"ContainerStarted","Data":"06807aaca02285c050d3dc086c73f6d19f13aed14346e26491de0a7d9c5a091c"} Jul 15 20:07:05.258407 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:05.258314 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:06.878133 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:06.878097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" event={"ID":"a75c5e33-236f-4726-87d4-f3edf06d437b","Type":"ContainerStarted","Data":"7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb"} Jul 15 20:07:06.878535 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:06.878186 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:06.908434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:06.908380 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" podStartSLOduration=2.173149988 podStartE2EDuration="3.908364671s" totalImagesPullingTime="1.735214683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:06.90767429 +0000 UTC m=+503.617240541" watchObservedRunningTime="2026-07-15 20:07:06.908364671 +0000 UTC m=+503.617930924" Jul 15 20:07:09.417273 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.417234 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms"] Jul 15 20:07:09.421806 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.421780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.424650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.424623 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-db22h\"" Jul 15 20:07:09.436430 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.436397 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms"] Jul 15 20:07:09.546123 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.546088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.546123 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.546131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hwc\" (UniqueName: \"kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.646762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.646714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.646762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.646763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hwc\" (UniqueName: \"kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.647178 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.647157 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.656442 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.656405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hwc\" (UniqueName: \"kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.732625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.732524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:09.873581 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.873555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms"] Jul 15 20:07:09.875858 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:09.875832 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4169bc_0a02_4ac0_a363_b53831a13796.slice/crio-804c386f20799287ef85dd6654140a8051d1c0a5aca2e5fc9f1439fbbb0020b6 WatchSource:0}: Error finding container 804c386f20799287ef85dd6654140a8051d1c0a5aca2e5fc9f1439fbbb0020b6: Status 404 returned error can't find the container with id 804c386f20799287ef85dd6654140a8051d1c0a5aca2e5fc9f1439fbbb0020b6 Jul 15 20:07:09.891134 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:09.891097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" event={"ID":"cb4169bc-0a02-4ac0-a363-b53831a13796","Type":"ContainerStarted","Data":"804c386f20799287ef85dd6654140a8051d1c0a5aca2e5fc9f1439fbbb0020b6"} Jul 15 20:07:10.184211 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:10.184175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:10.362959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:10.362922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:11.890048 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:11.889988 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:13.007036 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:13.006980 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:13.670886 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:13.670856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:14.455284 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:14.455240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:16.480585 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:16.480547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:16.926148 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:16.926114 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" event={"ID":"cb4169bc-0a02-4ac0-a363-b53831a13796","Type":"ContainerStarted","Data":"3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca"} Jul 15 20:07:16.926345 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:16.926240 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:16.955246 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:16.955185 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" podStartSLOduration=1.191312451 podStartE2EDuration="7.955164482s" totalImagesPullingTime="6.763852031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:16.955122281 +0000 UTC m=+513.664688534" watchObservedRunningTime="2026-07-15 20:07:16.955164482 +0000 UTC m=+513.664730734" Jul 15 20:07:17.884357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:17.884311 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:18.271825 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:18.271739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:18.506357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:18.506321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:18.864110 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:18.864077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:20.157600 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.157560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8"] Jul 15 20:07:20.160030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.159995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.163449 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.163423 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 20:07:20.163449 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.163441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fhct9\"" Jul 15 20:07:20.164127 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.164110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 20:07:20.172343 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.172316 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8"] Jul 15 20:07:20.342368 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.342323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.342548 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.342380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.342548 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.342401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75gn6\" (UniqueName: \"kubernetes.io/projected/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-kube-api-access-75gn6\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.443807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.443714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.443807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.443769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.443996 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.443884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75gn6\" (UniqueName: \"kubernetes.io/projected/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-kube-api-access-75gn6\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.444448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.444424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.446445 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.446414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.457747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.457716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75gn6\" (UniqueName: \"kubernetes.io/projected/e7b4579e-b81c-41b9-9b3c-4f5793da39f1-kube-api-access-75gn6\") pod \"kuadrant-console-plugin-66b5774985-z4pc8\" (UID: \"e7b4579e-b81c-41b9-9b3c-4f5793da39f1\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.469438 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.469405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" Jul 15 20:07:20.631620 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.631587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8"] Jul 15 20:07:20.635191 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:20.635160 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b4579e_b81c_41b9_9b3c_4f5793da39f1.slice/crio-74036b0ae07dc703fba13c36b160b94454bcb6f9a69de2088b43664b126f0b33 WatchSource:0}: Error finding container 74036b0ae07dc703fba13c36b160b94454bcb6f9a69de2088b43664b126f0b33: Status 404 returned error can't find the container with id 74036b0ae07dc703fba13c36b160b94454bcb6f9a69de2088b43664b126f0b33 Jul 15 20:07:20.717494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.717404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:20.789321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.789291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:20.946389 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:20.946355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" event={"ID":"e7b4579e-b81c-41b9-9b3c-4f5793da39f1","Type":"ContainerStarted","Data":"74036b0ae07dc703fba13c36b160b94454bcb6f9a69de2088b43664b126f0b33"} Jul 15 20:07:21.617078 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:21.617044 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:21.717502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:21.717465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:27.924936 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:27.924893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:27.933560 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:27.933529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:28.044745 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:28.044703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:30.803194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.803152 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms"] Jul 15 20:07:30.803645 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.803403 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" containerName="manager" containerID="cri-o://3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca" gracePeriod=2 Jul 15 20:07:30.807318 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.807231 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms"] Jul 15 20:07:30.894318 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.894277 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc"] Jul 15 20:07:30.894542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.894512 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" containerName="manager" containerID="cri-o://7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb" gracePeriod=2 Jul 15 20:07:30.915592 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.915548 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc"] Jul 15 20:07:30.917617 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.917582 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:30.918315 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918285 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" containerName="manager" Jul 15 20:07:30.918447 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" containerName="manager" Jul 15 20:07:30.918447 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918429 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" containerName="manager" Jul 15 20:07:30.918447 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918437 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" containerName="manager" Jul 15 20:07:30.918589 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918522 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" containerName="manager" Jul 15 20:07:30.918589 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.918539 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" containerName="manager" Jul 15 20:07:30.920538 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.920518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:30.930745 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.930701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:30.930923 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.930877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl6v\" (UniqueName: \"kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:30.939763 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:30.939718 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a75c5e33-236f-4726-87d4-f3edf06d437b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-qrwrc\": pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:30.943197 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:30.943154 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:30.945966 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:30.945919 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:30.950075 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:30.950035 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:30.959430 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.959364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:30.974202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.974071 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9"] Jul 15 20:07:30.981583 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:30.981552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:31.003148 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.003113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9"] Jul 15 20:07:31.031500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.031462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:31.031750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.031549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl6v\" (UniqueName: \"kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:31.031750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.031585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x49h\" (UniqueName: \"kubernetes.io/projected/93af46a2-d6f3-444b-8844-2dcd6f82d622-kube-api-access-9x49h\") pod \"limitador-operator-controller-manager-859d7b78bd-6stj9\" (UID: \"93af46a2-d6f3-444b-8844-2dcd6f82d622\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:31.031992 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.031968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:31.038242 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:31.038203 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:31.041272 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:31.041230 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:31.049191 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.049148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl6v\" (UniqueName: \"kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zjfpl\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:31.132919 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.132872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x49h\" (UniqueName: \"kubernetes.io/projected/93af46a2-d6f3-444b-8844-2dcd6f82d622-kube-api-access-9x49h\") pod \"limitador-operator-controller-manager-859d7b78bd-6stj9\" (UID: \"93af46a2-d6f3-444b-8844-2dcd6f82d622\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:31.152023 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.151988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x49h\" (UniqueName: \"kubernetes.io/projected/93af46a2-d6f3-444b-8844-2dcd6f82d622-kube-api-access-9x49h\") pod \"limitador-operator-controller-manager-859d7b78bd-6stj9\" (UID: \"93af46a2-d6f3-444b-8844-2dcd6f82d622\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:31.269800 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.269755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:31.296144 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:31.296107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:32.808989 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.808945 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:07:32.826673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.826635 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.833749 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.833718 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:07:32.847523 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.847476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.847682 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.847593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bt6\" (UniqueName: \"kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.948566 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.948522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.948760 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.948618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bt6\" (UniqueName: \"kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.949067 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.948998 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:32.959142 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:32.959105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bt6\" (UniqueName: \"kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6\") pod \"kuadrant-operator-controller-manager-86878c769f-6lq48\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:33.141173 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:33.141136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:34.390392 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:34.390249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:34.516960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:34.516928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:34.615942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:34.615909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:36.954847 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:36.954807 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:37.167495 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:37.167444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:39.126204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:39.126167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:39.323974 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:39.323941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:39.748631 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:39.748596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:40.003716 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:40.003634 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:44.349651 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.349598 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:44.352937 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:44.352903 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:44.372589 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.372559 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:44.375399 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:44.375347 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:44.377548 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:44.377508 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:44.383426 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.383402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9"] Jul 15 20:07:44.385726 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:44.385698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93af46a2_d6f3_444b_8844_2dcd6f82d622.slice/crio-9498cfb5c502ee115938b3739c76035e674d82a47331657f9b4a045520ae4046 WatchSource:0}: Error finding container 9498cfb5c502ee115938b3739c76035e674d82a47331657f9b4a045520ae4046: Status 404 returned error can't find the container with id 9498cfb5c502ee115938b3739c76035e674d82a47331657f9b4a045520ae4046 Jul 15 20:07:44.456359 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.456331 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume\") pod \"cb4169bc-0a02-4ac0-a363-b53831a13796\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " Jul 15 20:07:44.456481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.456378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4hwc\" (UniqueName: \"kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc\") pod \"cb4169bc-0a02-4ac0-a363-b53831a13796\" (UID: \"cb4169bc-0a02-4ac0-a363-b53831a13796\") " Jul 15 20:07:44.456481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.456465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrpv\" (UniqueName: \"kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv\") pod \"a75c5e33-236f-4726-87d4-f3edf06d437b\" (UID: \"a75c5e33-236f-4726-87d4-f3edf06d437b\") " Jul 15 20:07:44.456914 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.456863 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb4169bc-0a02-4ac0-a363-b53831a13796" (UID: "cb4169bc-0a02-4ac0-a363-b53831a13796"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:07:44.458482 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.458457 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv" (OuterVolumeSpecName: "kube-api-access-kmrpv") pod "a75c5e33-236f-4726-87d4-f3edf06d437b" (UID: "a75c5e33-236f-4726-87d4-f3edf06d437b"). InnerVolumeSpecName "kube-api-access-kmrpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:07:44.458786 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.458764 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc" (OuterVolumeSpecName: "kube-api-access-l4hwc") pod "cb4169bc-0a02-4ac0-a363-b53831a13796" (UID: "cb4169bc-0a02-4ac0-a363-b53831a13796"). InnerVolumeSpecName "kube-api-access-l4hwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:07:44.558059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.557989 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrpv\" (UniqueName: \"kubernetes.io/projected/a75c5e33-236f-4726-87d4-f3edf06d437b-kube-api-access-kmrpv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:07:44.558059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.558052 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb4169bc-0a02-4ac0-a363-b53831a13796-extensions-socket-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:07:44.558059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.558062 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hwc\" (UniqueName: \"kubernetes.io/projected/cb4169bc-0a02-4ac0-a363-b53831a13796-kube-api-access-l4hwc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:07:44.622204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.622170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:07:44.625773 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:44.625749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:44.626957 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:44.626925 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8084e85f_2eb8_48ef_8725_c438a577b6c6.slice/crio-c6f964b63bcfd0ad2f6a194c6c3e7587f8a64af854c56139c8f1ebb06f6be052 WatchSource:0}: Error finding container c6f964b63bcfd0ad2f6a194c6c3e7587f8a64af854c56139c8f1ebb06f6be052: Status 404 returned error can't find the container with id c6f964b63bcfd0ad2f6a194c6c3e7587f8a64af854c56139c8f1ebb06f6be052 Jul 15 20:07:44.627513 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:07:44.627487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac4d8b4_69c0_4bb1_a52f_0f7c2184630c.slice/crio-3ea899e973957c534dbcb435fac86fcbbe926e3e89a7415abda93691da18abc9 WatchSource:0}: Error finding container 3ea899e973957c534dbcb435fac86fcbbe926e3e89a7415abda93691da18abc9: Status 404 returned error can't find the container with id 3ea899e973957c534dbcb435fac86fcbbe926e3e89a7415abda93691da18abc9 Jul 15 20:07:45.080807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.080766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" event={"ID":"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c","Type":"ContainerStarted","Data":"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88"} Jul 15 20:07:45.081053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.080867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" event={"ID":"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c","Type":"ContainerStarted","Data":"3ea899e973957c534dbcb435fac86fcbbe926e3e89a7415abda93691da18abc9"} Jul 15 20:07:45.081053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.080902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:45.082157 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.082130 2576 generic.go:356] "Generic (PLEG): container finished" podID="cb4169bc-0a02-4ac0-a363-b53831a13796" containerID="3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca" exitCode=0 Jul 15 20:07:45.082268 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.082188 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:45.082268 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.082214 2576 scope.go:117] "RemoveContainer" containerID="3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca" Jul 15 20:07:45.088128 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.088089 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:45.089744 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.089711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" event={"ID":"8084e85f-2eb8-48ef-8725-c438a577b6c6","Type":"ContainerStarted","Data":"c97e6732b25edccaea1e239a22969b586af162999ec3afacf7fe0d216e0186f4"} Jul 15 20:07:45.089885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.089773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" event={"ID":"8084e85f-2eb8-48ef-8725-c438a577b6c6","Type":"ContainerStarted","Data":"c6f964b63bcfd0ad2f6a194c6c3e7587f8a64af854c56139c8f1ebb06f6be052"} Jul 15 20:07:45.090309 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.090278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:45.091027 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.090981 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:45.092456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.092433 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" event={"ID":"93af46a2-d6f3-444b-8844-2dcd6f82d622","Type":"ContainerStarted","Data":"5dff45162eaaa2db555d49d7a50231ea4418593278814598575c24ade2776da7"} Jul 15 20:07:45.092568 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.092480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" event={"ID":"93af46a2-d6f3-444b-8844-2dcd6f82d622","Type":"ContainerStarted","Data":"9498cfb5c502ee115938b3739c76035e674d82a47331657f9b4a045520ae4046"} Jul 15 20:07:45.092644 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.092583 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:45.093281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.093259 2576 scope.go:117] "RemoveContainer" containerID="3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca" Jul 15 20:07:45.093569 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.093552 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca\": container with ID starting with 3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca not found: ID does not exist" containerID="3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca" Jul 15 20:07:45.093622 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.093579 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca"} err="failed to get container status \"3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca\": rpc error: code = NotFound desc = could not find container \"3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca\": container with ID starting with 3a5b8d6c0ed2ffcf3bffb650ee7eb57726122495ccc80af686c4a4835c088eca not found: ID does not exist" Jul 15 20:07:45.094125 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.093966 2576 generic.go:356] "Generic (PLEG): container finished" podID="a75c5e33-236f-4726-87d4-f3edf06d437b" containerID="7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb" exitCode=0 Jul 15 20:07:45.094125 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.094043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:45.094125 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.094066 2576 scope.go:117] "RemoveContainer" containerID="7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb" Jul 15 20:07:45.096062 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.096035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" event={"ID":"e7b4579e-b81c-41b9-9b3c-4f5793da39f1","Type":"ContainerStarted","Data":"2b1c925b61054c958c5f42adbbbb0d8bc87b93befa3fd495c80a3f57247cd2d3"} Jul 15 20:07:45.102715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.102695 2576 scope.go:117] "RemoveContainer" containerID="7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb" Jul 15 20:07:45.102993 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.102974 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb\": container with ID starting with 7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb not found: ID does not exist" containerID="7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb" Jul 15 20:07:45.103072 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.103031 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb"} err="failed to get container status \"7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb\": rpc error: code = NotFound desc = could not find container \"7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb\": container with ID starting with 7da7534b644bd527b765504f0f3f8e08d6c841dc19ebedc806e444809d630ebb not found: ID does not exist" Jul 15 20:07:45.115372 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.115318 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" podStartSLOduration=15.115299678 podStartE2EDuration="15.115299678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:45.112157694 +0000 UTC m=+541.821723946" watchObservedRunningTime="2026-07-15 20:07:45.115299678 +0000 UTC m=+541.824865931" Jul 15 20:07:45.136887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.136835 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" podStartSLOduration=13.136817778 podStartE2EDuration="13.136817778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:45.135263827 +0000 UTC m=+541.844830079" watchObservedRunningTime="2026-07-15 20:07:45.136817778 +0000 UTC m=+541.846384030" Jul 15 20:07:45.156999 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.156955 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qrwrc\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qrwrc" Jul 15 20:07:45.157558 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.157514 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" podStartSLOduration=15.157498183 podStartE2EDuration="15.157498183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:45.154722337 +0000 UTC m=+541.864288589" watchObservedRunningTime="2026-07-15 20:07:45.157498183 +0000 UTC m=+541.867064434" Jul 15 20:07:45.183260 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:45.183217 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t55ms\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t55ms" Jul 15 20:07:45.850681 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.850641 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75c5e33-236f-4726-87d4-f3edf06d437b" path="/var/lib/kubelet/pods/a75c5e33-236f-4726-87d4-f3edf06d437b/volumes" Jul 15 20:07:45.851092 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:45.851001 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4169bc-0a02-4ac0-a363-b53831a13796" path="/var/lib/kubelet/pods/cb4169bc-0a02-4ac0-a363-b53831a13796/volumes" Jul 15 20:07:47.455695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:47.455660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:47.567859 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:47.567823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:53.719749 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:53.719705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:53.851995 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:53.851954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:54.067217 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:54.067128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:54.162362 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:54.162323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:07:56.105751 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:56.105714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6stj9" Jul 15 20:07:56.106146 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:56.105790 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:56.133866 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:56.133812 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z4pc8" podStartSLOduration=12.520961084 podStartE2EDuration="36.13379395s" totalImagesPullingTime="23.612832866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:07:45.180590966 +0000 UTC m=+541.890157219" watchObservedRunningTime="2026-07-15 20:07:56.13379395 +0000 UTC m=+552.843360230" Jul 15 20:07:57.111369 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.111337 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:07:57.177695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.177658 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:57.177900 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.177874 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" podUID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" containerName="manager" containerID="cri-o://b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88" gracePeriod=10 Jul 15 20:07:57.425183 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.425158 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:57.571542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.571506 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume\") pod \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " Jul 15 20:07:57.571714 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.571556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdl6v\" (UniqueName: \"kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v\") pod \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\" (UID: \"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c\") " Jul 15 20:07:57.571937 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.571899 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" (UID: "cac4d8b4-69c0-4bb1-a52f-0f7c2184630c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:07:57.573742 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.573709 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v" (OuterVolumeSpecName: "kube-api-access-hdl6v") pod "cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" (UID: "cac4d8b4-69c0-4bb1-a52f-0f7c2184630c"). InnerVolumeSpecName "kube-api-access-hdl6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:07:57.672692 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.672587 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-extensions-socket-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:07:57.672692 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:57.672631 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdl6v\" (UniqueName: \"kubernetes.io/projected/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c-kube-api-access-hdl6v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:07:58.162789 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.162749 2576 generic.go:356] "Generic (PLEG): container finished" podID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" containerID="b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88" exitCode=0 Jul 15 20:07:58.163207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.162802 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" Jul 15 20:07:58.163207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.162830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" event={"ID":"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c","Type":"ContainerDied","Data":"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88"} Jul 15 20:07:58.163207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.162884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl" event={"ID":"cac4d8b4-69c0-4bb1-a52f-0f7c2184630c","Type":"ContainerDied","Data":"3ea899e973957c534dbcb435fac86fcbbe926e3e89a7415abda93691da18abc9"} Jul 15 20:07:58.163207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.162908 2576 scope.go:117] "RemoveContainer" containerID="b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88" Jul 15 20:07:58.171280 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.171257 2576 scope.go:117] "RemoveContainer" containerID="b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88" Jul 15 20:07:58.171612 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:07:58.171594 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88\": container with ID starting with b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88 not found: ID does not exist" containerID="b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88" Jul 15 20:07:58.171670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.171625 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88"} err="failed to get container status \"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88\": rpc error: code = NotFound desc = could not find container \"b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88\": container with ID starting with b6e42c5de2097c2b591c14cc3d4c02eec6a74fae2d21180922d3fc59173eef88 not found: ID does not exist" Jul 15 20:07:58.183457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.183425 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:58.186855 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:58.186830 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zjfpl"] Jul 15 20:07:59.847894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:07:59.847859 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" path="/var/lib/kubelet/pods/cac4d8b4-69c0-4bb1-a52f-0f7c2184630c/volumes" Jul 15 20:08:14.015586 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.015550 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4"] Jul 15 20:08:14.016028 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.015853 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" containerName="manager" Jul 15 20:08:14.016028 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.015864 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" containerName="manager" Jul 15 20:08:14.016028 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.015942 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac4d8b4-69c0-4bb1-a52f-0f7c2184630c" containerName="manager" Jul 15 20:08:14.017768 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.017749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.024844 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.024821 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b6lt2\"" Jul 15 20:08:14.054616 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.054581 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4"] Jul 15 20:08:14.060747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.060709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:14.115725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.115725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f6fb60-49fc-42c0-8328-bac5525298d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.115951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.115951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.115951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.115951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-kube-api-access-nknl4\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.116151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.115990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.116151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.116083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.116151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.116119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.216976 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.216936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f6fb60-49fc-42c0-8328-bac5525298d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-kube-api-access-nknl4\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217655 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217655 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217737 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.217858 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.217834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f6fb60-49fc-42c0-8328-bac5525298d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.219385 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.219367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.219971 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.219933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.234250 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.234208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-kube-api-access-nknl4\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.234250 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.234236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f6fb60-49fc-42c0-8328-bac5525298d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-8xbb4\" (UID: \"e0f6fb60-49fc-42c0-8328-bac5525298d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.310874 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.310786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:14.328991 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.328954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:14.497807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.497774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4"] Jul 15 20:08:14.501376 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:14.501344 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f6fb60_49fc_42c0_8328_bac5525298d1.slice/crio-43eb69268e538e0b04f7524e38c0faee214df844bf7f9bcbc138a317715f840d WatchSource:0}: Error finding container 43eb69268e538e0b04f7524e38c0faee214df844bf7f9bcbc138a317715f840d: Status 404 returned error can't find the container with id 43eb69268e538e0b04f7524e38c0faee214df844bf7f9bcbc138a317715f840d Jul 15 20:08:14.503415 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.503383 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:08:14.503506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.503448 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:08:14.503506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:14.503481 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:08:15.252887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:15.252855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" event={"ID":"e0f6fb60-49fc-42c0-8328-bac5525298d1","Type":"ContainerStarted","Data":"059435881cc4eb41e88d8eddc5abc5ab2bb59aaf3ebbee87663e56554008e640"} Jul 15 20:08:15.253298 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:15.252913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" event={"ID":"e0f6fb60-49fc-42c0-8328-bac5525298d1","Type":"ContainerStarted","Data":"43eb69268e538e0b04f7524e38c0faee214df844bf7f9bcbc138a317715f840d"} Jul 15 20:08:15.295425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:15.295375 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" podStartSLOduration=2.295360613 podStartE2EDuration="2.295360613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:15.29514956 +0000 UTC m=+572.004715813" watchObservedRunningTime="2026-07-15 20:08:15.295360613 +0000 UTC m=+572.004926866" Jul 15 20:08:15.329463 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:15.329415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:15.334584 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:15.334557 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:16.257323 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.257285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:16.258478 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.258456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:08:16.739477 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.739442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:16.768632 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.768598 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:16.798746 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.798708 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:16.798958 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.798816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:16.804885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.804846 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 20:08:16.869619 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.869582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:16.925929 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.925898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:16.937878 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.937839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:16.938105 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:16.938088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v7b6\" (UniqueName: \"kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.039111 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.038998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v7b6\" (UniqueName: \"kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.039111 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.039091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.039799 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.039774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.063638 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.063601 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v7b6\" (UniqueName: \"kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6\") pod \"limitador-limitador-64c94fd64f-458rp\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.111238 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.111203 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:17.172687 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.172651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:17.343198 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:17.343169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:17.345667 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:17.345640 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40758f06_72fe_4544_a92e_f4ac2d630ae6.slice/crio-2eb6437d8ec4d61604639895b1c84114e644743da13604471d393711a2755c99 WatchSource:0}: Error finding container 2eb6437d8ec4d61604639895b1c84114e644743da13604471d393711a2755c99: Status 404 returned error can't find the container with id 2eb6437d8ec4d61604639895b1c84114e644743da13604471d393711a2755c99 Jul 15 20:08:18.027077 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.027039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:18.273359 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.273309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" event={"ID":"40758f06-72fe-4544-a92e-f4ac2d630ae6","Type":"ContainerStarted","Data":"2eb6437d8ec4d61604639895b1c84114e644743da13604471d393711a2755c99"} Jul 15 20:08:18.331676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.331623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:18.895653 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.895610 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:18.900024 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.899981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:18.906377 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.906347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mx56w\"" Jul 15 20:08:18.913380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:18.913350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:19.056195 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:19.056157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rggp\" (UniqueName: \"kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp\") pod \"authorino-75d556bd77-pml9q\" (UID: \"9373dbec-a46f-46bf-9f3d-fdea4b8decfe\") " pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:19.157344 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:19.157245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rggp\" (UniqueName: \"kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp\") pod \"authorino-75d556bd77-pml9q\" (UID: \"9373dbec-a46f-46bf-9f3d-fdea4b8decfe\") " pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:19.171650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:19.171613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rggp\" (UniqueName: \"kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp\") pod \"authorino-75d556bd77-pml9q\" (UID: \"9373dbec-a46f-46bf-9f3d-fdea4b8decfe\") " pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:19.211486 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:19.211442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:20.252150 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:20.252120 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:20.254858 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:20.254827 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9373dbec_a46f_46bf_9f3d_fdea4b8decfe.slice/crio-b23360aca9706db02bf6db4c652370cbab18c7a0f85345b85fbe3b950fcc972a WatchSource:0}: Error finding container b23360aca9706db02bf6db4c652370cbab18c7a0f85345b85fbe3b950fcc972a: Status 404 returned error can't find the container with id b23360aca9706db02bf6db4c652370cbab18c7a0f85345b85fbe3b950fcc972a Jul 15 20:08:20.285379 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:20.285344 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" event={"ID":"40758f06-72fe-4544-a92e-f4ac2d630ae6","Type":"ContainerStarted","Data":"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1"} Jul 15 20:08:20.285555 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:20.285456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:20.286736 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:20.286712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pml9q" event={"ID":"9373dbec-a46f-46bf-9f3d-fdea4b8decfe","Type":"ContainerStarted","Data":"b23360aca9706db02bf6db4c652370cbab18c7a0f85345b85fbe3b950fcc972a"} Jul 15 20:08:20.313875 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:20.313821 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" podStartSLOduration=1.5144436589999999 podStartE2EDuration="4.313802372s" totalImagesPullingTime="2.799358713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:20.309879847 +0000 UTC m=+577.019446098" watchObservedRunningTime="2026-07-15 20:08:20.313802372 +0000 UTC m=+577.023368624" Jul 15 20:08:23.304892 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:23.304849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pml9q" event={"ID":"9373dbec-a46f-46bf-9f3d-fdea4b8decfe","Type":"ContainerStarted","Data":"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988"} Jul 15 20:08:31.294821 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:31.294793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:31.317763 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:31.317705 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pml9q" podStartSLOduration=10.487941156 podStartE2EDuration="13.31768852s" totalImagesPullingTime="2.829747364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:23.334295066 +0000 UTC m=+580.043861317" watchObservedRunningTime="2026-07-15 20:08:31.31768852 +0000 UTC m=+588.027254786" Jul 15 20:08:32.166136 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.166094 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:32.166347 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.166321 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" podUID="40758f06-72fe-4544-a92e-f4ac2d630ae6" containerName="limitador" containerID="cri-o://be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1" gracePeriod=30 Jul 15 20:08:32.712637 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.712608 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:32.772442 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.772358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v7b6\" (UniqueName: \"kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6\") pod \"40758f06-72fe-4544-a92e-f4ac2d630ae6\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " Jul 15 20:08:32.772442 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.772402 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file\") pod \"40758f06-72fe-4544-a92e-f4ac2d630ae6\" (UID: \"40758f06-72fe-4544-a92e-f4ac2d630ae6\") " Jul 15 20:08:32.772847 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.772820 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file" (OuterVolumeSpecName: "config-file") pod "40758f06-72fe-4544-a92e-f4ac2d630ae6" (UID: "40758f06-72fe-4544-a92e-f4ac2d630ae6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:08:32.774513 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.774488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6" (OuterVolumeSpecName: "kube-api-access-6v7b6") pod "40758f06-72fe-4544-a92e-f4ac2d630ae6" (UID: "40758f06-72fe-4544-a92e-f4ac2d630ae6"). InnerVolumeSpecName "kube-api-access-6v7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:32.873296 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.873256 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v7b6\" (UniqueName: \"kubernetes.io/projected/40758f06-72fe-4544-a92e-f4ac2d630ae6-kube-api-access-6v7b6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:08:32.873296 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:32.873290 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/40758f06-72fe-4544-a92e-f4ac2d630ae6-config-file\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:08:33.353963 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.353925 2576 generic.go:356] "Generic (PLEG): container finished" podID="40758f06-72fe-4544-a92e-f4ac2d630ae6" containerID="be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1" exitCode=0 Jul 15 20:08:33.354194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.353972 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" Jul 15 20:08:33.354194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.353999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" event={"ID":"40758f06-72fe-4544-a92e-f4ac2d630ae6","Type":"ContainerDied","Data":"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1"} Jul 15 20:08:33.354194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.354090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-458rp" event={"ID":"40758f06-72fe-4544-a92e-f4ac2d630ae6","Type":"ContainerDied","Data":"2eb6437d8ec4d61604639895b1c84114e644743da13604471d393711a2755c99"} Jul 15 20:08:33.354194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.354119 2576 scope.go:117] "RemoveContainer" containerID="be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1" Jul 15 20:08:33.363252 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.363228 2576 scope.go:117] "RemoveContainer" containerID="be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1" Jul 15 20:08:33.363569 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:08:33.363545 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1\": container with ID starting with be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1 not found: ID does not exist" containerID="be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1" Jul 15 20:08:33.363634 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.363584 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1"} err="failed to get container status \"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1\": rpc error: code = NotFound desc = could not find container \"be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1\": container with ID starting with be4add8d7e3114cb95293999876f0da97bf55b28d30ef5af6866922f5ce530d1 not found: ID does not exist" Jul 15 20:08:33.379083 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.379042 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:33.384025 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.383983 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-458rp"] Jul 15 20:08:33.848249 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:33.848212 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40758f06-72fe-4544-a92e-f4ac2d630ae6" path="/var/lib/kubelet/pods/40758f06-72fe-4544-a92e-f4ac2d630ae6/volumes" Jul 15 20:08:37.864255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:37.864218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:42.688021 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:42.687982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:42.788113 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:42.788079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:43.366517 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.366482 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw"] Jul 15 20:08:43.366842 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.366829 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40758f06-72fe-4544-a92e-f4ac2d630ae6" containerName="limitador" Jul 15 20:08:43.366892 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.366844 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40758f06-72fe-4544-a92e-f4ac2d630ae6" containerName="limitador" Jul 15 20:08:43.366926 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.366901 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40758f06-72fe-4544-a92e-f4ac2d630ae6" containerName="limitador" Jul 15 20:08:43.369996 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.369976 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.373652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.373622 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:08:43.373652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.373647 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 20:08:43.373863 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.373625 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 20:08:43.374687 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.374672 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qtjv6\"" Jul 15 20:08:43.379910 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.379880 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw"] Jul 15 20:08:43.465895 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.465861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqdb\" (UniqueName: \"kubernetes.io/projected/4cfed204-d523-4d95-95da-0762ef9dbb27-kube-api-access-4nqdb\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.466112 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.465910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4cfed204-d523-4d95-95da-0762ef9dbb27-data\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.566789 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.566748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqdb\" (UniqueName: \"kubernetes.io/projected/4cfed204-d523-4d95-95da-0762ef9dbb27-kube-api-access-4nqdb\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.566948 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.566801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4cfed204-d523-4d95-95da-0762ef9dbb27-data\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.567179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.567162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4cfed204-d523-4d95-95da-0762ef9dbb27-data\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.576750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.576714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqdb\" (UniqueName: \"kubernetes.io/projected/4cfed204-d523-4d95-95da-0762ef9dbb27-kube-api-access-4nqdb\") pod \"postgres-69fb4594cf-tbwqw\" (UID: \"4cfed204-d523-4d95-95da-0762ef9dbb27\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.684205 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.684117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qtjv6\"" Jul 15 20:08:43.691197 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.691174 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:43.696267 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.696237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:08:43.698860 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.698837 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:08:43.845606 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:43.845573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfed204_d523_4d95_95da_0762ef9dbb27.slice/crio-253a45fe96fadd3c8833f4313814de2212e59e2826836209f7fb211bbc4ace55 WatchSource:0}: Error finding container 253a45fe96fadd3c8833f4313814de2212e59e2826836209f7fb211bbc4ace55: Status 404 returned error can't find the container with id 253a45fe96fadd3c8833f4313814de2212e59e2826836209f7fb211bbc4ace55 Jul 15 20:08:43.849729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:43.849705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw"] Jul 15 20:08:44.406293 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:44.406254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" event={"ID":"4cfed204-d523-4d95-95da-0762ef9dbb27","Type":"ContainerStarted","Data":"253a45fe96fadd3c8833f4313814de2212e59e2826836209f7fb211bbc4ace55"} Jul 15 20:08:45.310987 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:45.310915 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:49.074589 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:49.074563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 20:08:49.435871 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:49.435833 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" event={"ID":"4cfed204-d523-4d95-95da-0762ef9dbb27","Type":"ContainerStarted","Data":"878210da09c2f5b85ab4c178b28e6b33ee70abfec352a145b5be686ea69c8163"} Jul 15 20:08:49.436087 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:49.435979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:49.455374 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:49.455307 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" podStartSLOduration=1.230874814 podStartE2EDuration="6.455290883s" totalImagesPullingTime="5.224416069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:49.453379175 +0000 UTC m=+606.162945428" watchObservedRunningTime="2026-07-15 20:08:49.455290883 +0000 UTC m=+606.164857135" Jul 15 20:08:55.471293 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:55.471263 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tbwqw" Jul 15 20:08:55.808388 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:55.808276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:55.910576 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:55.910544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:08:56.070733 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.070652 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:56.077890 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.077862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:56.082634 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.082594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:56.175557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.175519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngn4x\" (UniqueName: \"kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x\") pod \"authorino-5d4b67c446-44nt7\" (UID: \"9c254e69-55cf-47d8-8611-1d2ab6525fbb\") " pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:56.276379 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.276338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngn4x\" (UniqueName: \"kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x\") pod \"authorino-5d4b67c446-44nt7\" (UID: \"9c254e69-55cf-47d8-8611-1d2ab6525fbb\") " pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:56.286144 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.286108 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngn4x\" (UniqueName: \"kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x\") pod \"authorino-5d4b67c446-44nt7\" (UID: \"9c254e69-55cf-47d8-8611-1d2ab6525fbb\") " pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:56.319841 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.319796 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:56.320121 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.320104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:56.341941 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.341905 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5897fd55c9-spwjk"] Jul 15 20:08:56.345222 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.345198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.355328 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.355289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5897fd55c9-spwjk"] Jul 15 20:08:56.443099 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.443067 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5897fd55c9-spwjk"] Jul 15 20:08:56.443368 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:08:56.443348 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zh8x6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5897fd55c9-spwjk" podUID="d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b" Jul 15 20:08:56.467921 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.467891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.473835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.473810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.478644 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.478619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8x6\" (UniqueName: \"kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6\") pod \"authorino-5897fd55c9-spwjk\" (UID: \"d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b\") " pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.482911 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.482889 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:56.485675 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:56.485646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c254e69_55cf_47d8_8611_1d2ab6525fbb.slice/crio-ab81fffab0ca1b3e3a5f78d9c5401c7fdb7c23299584a1f8e31242818effbd28 WatchSource:0}: Error finding container ab81fffab0ca1b3e3a5f78d9c5401c7fdb7c23299584a1f8e31242818effbd28: Status 404 returned error can't find the container with id ab81fffab0ca1b3e3a5f78d9c5401c7fdb7c23299584a1f8e31242818effbd28 Jul 15 20:08:56.503542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.503502 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:08:56.505849 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.505823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.508587 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.508562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 20:08:56.520061 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.520033 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:08:56.579342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.579310 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.579497 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.579355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8x6\" (UniqueName: \"kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6\") pod \"authorino-5897fd55c9-spwjk\" (UID: \"d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b\") " pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.579497 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.579454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmjd\" (UniqueName: \"kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.591450 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.591419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8x6\" (UniqueName: \"kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6\") pod \"authorino-5897fd55c9-spwjk\" (UID: \"d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b\") " pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:56.680668 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.680632 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8x6\" (UniqueName: \"kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6\") pod \"d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b\" (UID: \"d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b\") " Jul 15 20:08:56.680833 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.680803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.680882 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.680869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmjd\" (UniqueName: \"kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.682808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.682780 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6" (OuterVolumeSpecName: "kube-api-access-zh8x6") pod "d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b" (UID: "d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b"). InnerVolumeSpecName "kube-api-access-zh8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:56.683204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.683181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.693037 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.692986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmjd\" (UniqueName: \"kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd\") pod \"authorino-8467fbf95-cx97c\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.781984 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.781955 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8x6\" (UniqueName: \"kubernetes.io/projected/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b-kube-api-access-zh8x6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:08:56.816478 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.816450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:08:56.978457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:56.978431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:08:56.980232 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:08:56.980207 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75811df7_fdd5_4d21_bf36_97c8d12bf5ed.slice/crio-961f7933f07e062851b856f19b81342caaa1ebbb5840d894bcc26fe9033832f1 WatchSource:0}: Error finding container 961f7933f07e062851b856f19b81342caaa1ebbb5840d894bcc26fe9033832f1: Status 404 returned error can't find the container with id 961f7933f07e062851b856f19b81342caaa1ebbb5840d894bcc26fe9033832f1 Jul 15 20:08:57.475314 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.475273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8467fbf95-cx97c" event={"ID":"75811df7-fdd5-4d21-bf36-97c8d12bf5ed","Type":"ContainerStarted","Data":"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95"} Jul 15 20:08:57.475815 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.475328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8467fbf95-cx97c" event={"ID":"75811df7-fdd5-4d21-bf36-97c8d12bf5ed","Type":"ContainerStarted","Data":"961f7933f07e062851b856f19b81342caaa1ebbb5840d894bcc26fe9033832f1"} Jul 15 20:08:57.476885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.476865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5897fd55c9-spwjk" Jul 15 20:08:57.477047 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.476915 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-44nt7" podUID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" containerName="authorino" containerID="cri-o://e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0" gracePeriod=30 Jul 15 20:08:57.477047 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.476861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-44nt7" event={"ID":"9c254e69-55cf-47d8-8611-1d2ab6525fbb","Type":"ContainerStarted","Data":"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0"} Jul 15 20:08:57.477047 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.476972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-44nt7" event={"ID":"9c254e69-55cf-47d8-8611-1d2ab6525fbb","Type":"ContainerStarted","Data":"ab81fffab0ca1b3e3a5f78d9c5401c7fdb7c23299584a1f8e31242818effbd28"} Jul 15 20:08:57.498198 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.498143 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8467fbf95-cx97c" podStartSLOduration=1.087279092 podStartE2EDuration="1.498124421s" totalImagesPullingTime="410.845329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:57.496764103 +0000 UTC m=+614.206330355" watchObservedRunningTime="2026-07-15 20:08:57.498124421 +0000 UTC m=+614.207690673" Jul 15 20:08:57.519034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.518954 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-44nt7" podStartSLOduration=1.239271147 podStartE2EDuration="1.518938409s" totalImagesPullingTime="279.667262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:08:57.517427506 +0000 UTC m=+614.226993759" watchObservedRunningTime="2026-07-15 20:08:57.518938409 +0000 UTC m=+614.228504661" Jul 15 20:08:57.540454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.540401 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:57.540651 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.540622 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pml9q" podUID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" containerName="authorino" containerID="cri-o://aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988" gracePeriod=30 Jul 15 20:08:57.555556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.555517 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5897fd55c9-spwjk"] Jul 15 20:08:57.562215 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.562156 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5897fd55c9-spwjk"] Jul 15 20:08:57.786345 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.786319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:57.821255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.821228 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:57.848905 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.848869 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b" path="/var/lib/kubelet/pods/d2e8c1a9-7fd8-416a-86f6-237f8c2cea2b/volumes" Jul 15 20:08:57.891477 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.891442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rggp\" (UniqueName: \"kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp\") pod \"9373dbec-a46f-46bf-9f3d-fdea4b8decfe\" (UID: \"9373dbec-a46f-46bf-9f3d-fdea4b8decfe\") " Jul 15 20:08:57.891646 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.891504 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngn4x\" (UniqueName: \"kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x\") pod \"9c254e69-55cf-47d8-8611-1d2ab6525fbb\" (UID: \"9c254e69-55cf-47d8-8611-1d2ab6525fbb\") " Jul 15 20:08:57.893669 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.893632 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp" (OuterVolumeSpecName: "kube-api-access-9rggp") pod "9373dbec-a46f-46bf-9f3d-fdea4b8decfe" (UID: "9373dbec-a46f-46bf-9f3d-fdea4b8decfe"). InnerVolumeSpecName "kube-api-access-9rggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:57.893669 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.893671 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x" (OuterVolumeSpecName: "kube-api-access-ngn4x") pod "9c254e69-55cf-47d8-8611-1d2ab6525fbb" (UID: "9c254e69-55cf-47d8-8611-1d2ab6525fbb"). InnerVolumeSpecName "kube-api-access-ngn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:08:57.992313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.992259 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rggp\" (UniqueName: \"kubernetes.io/projected/9373dbec-a46f-46bf-9f3d-fdea4b8decfe-kube-api-access-9rggp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:08:57.992313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:57.992307 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngn4x\" (UniqueName: \"kubernetes.io/projected/9c254e69-55cf-47d8-8611-1d2ab6525fbb-kube-api-access-ngn4x\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:08:58.483752 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.483710 2576 generic.go:356] "Generic (PLEG): container finished" podID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" containerID="e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0" exitCode=0 Jul 15 20:08:58.484179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.483761 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-44nt7" Jul 15 20:08:58.484179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.483794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-44nt7" event={"ID":"9c254e69-55cf-47d8-8611-1d2ab6525fbb","Type":"ContainerDied","Data":"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0"} Jul 15 20:08:58.484179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.483849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-44nt7" event={"ID":"9c254e69-55cf-47d8-8611-1d2ab6525fbb","Type":"ContainerDied","Data":"ab81fffab0ca1b3e3a5f78d9c5401c7fdb7c23299584a1f8e31242818effbd28"} Jul 15 20:08:58.484179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.483874 2576 scope.go:117] "RemoveContainer" containerID="e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0" Jul 15 20:08:58.485180 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.485125 2576 generic.go:356] "Generic (PLEG): container finished" podID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" containerID="aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988" exitCode=0 Jul 15 20:08:58.485268 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.485243 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pml9q" Jul 15 20:08:58.485352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.485268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pml9q" event={"ID":"9373dbec-a46f-46bf-9f3d-fdea4b8decfe","Type":"ContainerDied","Data":"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988"} Jul 15 20:08:58.485352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.485308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pml9q" event={"ID":"9373dbec-a46f-46bf-9f3d-fdea4b8decfe","Type":"ContainerDied","Data":"b23360aca9706db02bf6db4c652370cbab18c7a0f85345b85fbe3b950fcc972a"} Jul 15 20:08:58.492336 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.492314 2576 scope.go:117] "RemoveContainer" containerID="e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0" Jul 15 20:08:58.492656 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:08:58.492637 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0\": container with ID starting with e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0 not found: ID does not exist" containerID="e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0" Jul 15 20:08:58.492726 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.492664 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0"} err="failed to get container status \"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0\": rpc error: code = NotFound desc = could not find container \"e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0\": container with ID starting with e75411d1c9df73af86d42ca959dd3864e30ebf37d64c24432b5cf6efa2cbf7b0 not found: ID does not exist" Jul 15 20:08:58.492726 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.492680 2576 scope.go:117] "RemoveContainer" containerID="aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988" Jul 15 20:08:58.500575 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.500552 2576 scope.go:117] "RemoveContainer" containerID="aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988" Jul 15 20:08:58.500859 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:08:58.500839 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988\": container with ID starting with aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988 not found: ID does not exist" containerID="aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988" Jul 15 20:08:58.500913 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.500868 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988"} err="failed to get container status \"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988\": rpc error: code = NotFound desc = could not find container \"aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988\": container with ID starting with aaa15cf0159afc359bec5b1fe742b1f69b1a9c87be1cf68b0c5fe6fe062a2988 not found: ID does not exist" Jul 15 20:08:58.787407 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.787374 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:58.795920 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.795887 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-44nt7"] Jul 15 20:08:58.814573 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.814525 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:58.823750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:58.823717 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pml9q"] Jul 15 20:08:59.849539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:59.849508 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" path="/var/lib/kubelet/pods/9373dbec-a46f-46bf-9f3d-fdea4b8decfe/volumes" Jul 15 20:08:59.849961 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:08:59.849833 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" path="/var/lib/kubelet/pods/9c254e69-55cf-47d8-8611-1d2ab6525fbb/volumes" Jul 15 20:09:06.440070 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:06.439975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:07.113214 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:07.113186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:07.407976 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:07.407893 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:09.013565 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:09.013528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:09.072869 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:09.072833 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:11.835657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:11.835617 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:12.130877 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:12.130762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:12.218134 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:12.218099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:44.051177 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:44.051096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:45.779005 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.778967 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-77fnq"] Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779316 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" containerName="authorino" Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779329 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" containerName="authorino" Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779343 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" containerName="authorino" Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779349 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" containerName="authorino" Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779402 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9373dbec-a46f-46bf-9f3d-fdea4b8decfe" containerName="authorino" Jul 15 20:09:45.779661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.779411 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c254e69-55cf-47d8-8611-1d2ab6525fbb" containerName="authorino" Jul 15 20:09:45.783514 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.783486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:45.786545 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.786516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 20:09:45.786666 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.786635 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ccvv5\"" Jul 15 20:09:45.799701 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.799669 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-77fnq"] Jul 15 20:09:45.916972 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.916936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h25xf\" (UniqueName: \"kubernetes.io/projected/b5b47d8d-f9ff-4431-8210-a1663fc15d26-kube-api-access-h25xf\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:45.917157 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:45.916991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b47d8d-f9ff-4431-8210-a1663fc15d26-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.017783 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.017742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h25xf\" (UniqueName: \"kubernetes.io/projected/b5b47d8d-f9ff-4431-8210-a1663fc15d26-kube-api-access-h25xf\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.017783 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.017790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b47d8d-f9ff-4431-8210-a1663fc15d26-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.018489 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.018469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b47d8d-f9ff-4431-8210-a1663fc15d26-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.036714 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.036627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h25xf\" (UniqueName: \"kubernetes.io/projected/b5b47d8d-f9ff-4431-8210-a1663fc15d26-kube-api-access-h25xf\") pod \"payload-processing-f9b8d9b9b-77fnq\" (UID: \"b5b47d8d-f9ff-4431-8210-a1663fc15d26\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.090861 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.090817 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:46.094651 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.094620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:46.251507 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.251473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-77fnq"] Jul 15 20:09:46.253497 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:09:46.253452 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b47d8d_f9ff_4431_8210_a1663fc15d26.slice/crio-96b1f720afea4d3d29c9b8c574364af2b2a6ce3f0dcf06cb300a84cbb787b1de WatchSource:0}: Error finding container 96b1f720afea4d3d29c9b8c574364af2b2a6ce3f0dcf06cb300a84cbb787b1de: Status 404 returned error can't find the container with id 96b1f720afea4d3d29c9b8c574364af2b2a6ce3f0dcf06cb300a84cbb787b1de Jul 15 20:09:46.255584 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.255560 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:09:46.709295 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:46.709252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" event={"ID":"b5b47d8d-f9ff-4431-8210-a1663fc15d26","Type":"ContainerStarted","Data":"96b1f720afea4d3d29c9b8c574364af2b2a6ce3f0dcf06cb300a84cbb787b1de"} Jul 15 20:09:48.722640 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:48.722603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" event={"ID":"b5b47d8d-f9ff-4431-8210-a1663fc15d26","Type":"ContainerStarted","Data":"84642610da27d28f97fd20ac6c550253a6c3f3d6ab9846f736d59110b5743bfd"} Jul 15 20:09:48.723141 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:48.722692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:09:48.745961 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:48.745905 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" podStartSLOduration=2.182844669 podStartE2EDuration="3.745887357s" totalImagesPullingTime="1.563042688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:09:48.744295759 +0000 UTC m=+665.453862040" watchObservedRunningTime="2026-07-15 20:09:48.745887357 +0000 UTC m=+665.455453609" Jul 15 20:09:53.750216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.750181 2576 generic.go:356] "Generic (PLEG): container finished" podID="e0f6fb60-49fc-42c0-8328-bac5525298d1" containerID="059435881cc4eb41e88d8eddc5abc5ab2bb59aaf3ebbee87663e56554008e640" exitCode=0 Jul 15 20:09:53.750737 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.750261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" event={"ID":"e0f6fb60-49fc-42c0-8328-bac5525298d1","Type":"ContainerDied","Data":"059435881cc4eb41e88d8eddc5abc5ab2bb59aaf3ebbee87663e56554008e640"} Jul 15 20:09:53.750808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.750789 2576 scope.go:117] "RemoveContainer" containerID="059435881cc4eb41e88d8eddc5abc5ab2bb59aaf3ebbee87663e56554008e640" Jul 15 20:09:53.751862 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.751832 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:09:53.751956 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.751902 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:09:53.751956 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:53.751935 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:09:54.756736 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:54.756698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" event={"ID":"e0f6fb60-49fc-42c0-8328-bac5525298d1","Type":"ContainerStarted","Data":"7ca7c87f015e06f9b832241a71bc75bd5072d141c6d81734c11969f605faa8a7"} Jul 15 20:09:55.329377 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:55.329320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:09:55.329636 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:55.329612 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8xbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 20:09:55.329725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:55.329684 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" podUID="e0f6fb60-49fc-42c0-8328-bac5525298d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 20:09:55.805433 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:55.805191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:55.903885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:55.903845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:09:56.329835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:56.329795 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8xbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 20:09:56.330054 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:56.329857 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" podUID="e0f6fb60-49fc-42c0-8328-bac5525298d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 20:09:57.330169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:57.330136 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8xbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 20:09:57.330567 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:57.330190 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" podUID="e0f6fb60-49fc-42c0-8328-bac5525298d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 20:09:58.329968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:58.329932 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-8xbb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 15 20:09:58.330173 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:58.329994 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" podUID="e0f6fb60-49fc-42c0-8328-bac5525298d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 15 20:09:59.334167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:59.334136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:09:59.334533 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:59.334410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:09:59.335220 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:59.335205 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-8xbb4" Jul 15 20:09:59.729976 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:09:59.729890 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-77fnq" Jul 15 20:10:14.379520 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.379480 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq"] Jul 15 20:10:14.381651 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.381632 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.384616 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.384591 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 20:10:14.386043 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.385999 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 20:10:14.386139 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.386001 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rkl7w\"" Jul 15 20:10:14.386139 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.386002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 20:10:14.394858 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.394826 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq"] Jul 15 20:10:14.458779 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.458981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58ts\" (UniqueName: \"kubernetes.io/projected/8427d260-d1c2-4e8f-a3be-873f5efd57cc-kube-api-access-r58ts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.458981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.458981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.458981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.459197 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.458980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.494883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.494843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:14.559557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.559557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.559816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.559816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58ts\" (UniqueName: \"kubernetes.io/projected/8427d260-d1c2-4e8f-a3be-873f5efd57cc-kube-api-access-r58ts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.559816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.559816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.560055 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.559977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.560123 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.560070 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.560166 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.560127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.562036 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.562000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8427d260-d1c2-4e8f-a3be-873f5efd57cc-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.562307 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.562288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8427d260-d1c2-4e8f-a3be-873f5efd57cc-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.569894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.569860 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58ts\" (UniqueName: \"kubernetes.io/projected/8427d260-d1c2-4e8f-a3be-873f5efd57cc-kube-api-access-r58ts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq\" (UID: \"8427d260-d1c2-4e8f-a3be-873f5efd57cc\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.692819 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.692727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:14.840648 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.840618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq"] Jul 15 20:10:14.843023 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:10:14.842976 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8427d260_d1c2_4e8f_a3be_873f5efd57cc.slice/crio-7aa465996f001cb98d0dc1db12dba99778e8bc574216207fdc510e5bb75dfe4c WatchSource:0}: Error finding container 7aa465996f001cb98d0dc1db12dba99778e8bc574216207fdc510e5bb75dfe4c: Status 404 returned error can't find the container with id 7aa465996f001cb98d0dc1db12dba99778e8bc574216207fdc510e5bb75dfe4c Jul 15 20:10:14.851194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:14.851160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" event={"ID":"8427d260-d1c2-4e8f-a3be-873f5efd57cc","Type":"ContainerStarted","Data":"7aa465996f001cb98d0dc1db12dba99778e8bc574216207fdc510e5bb75dfe4c"} Jul 15 20:10:16.862311 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:16.862278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" event={"ID":"8427d260-d1c2-4e8f-a3be-873f5efd57cc","Type":"ContainerStarted","Data":"0abcd7979cae70d90a11b383f445f453d188985df30a9db0aca0de15caf8a84f"} Jul 15 20:10:16.885681 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:16.885622 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" podStartSLOduration=1.6125682399999999 podStartE2EDuration="2.8856031s" totalImagesPullingTime="1.27303486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:10:16.883332222 +0000 UTC m=+693.592898475" watchObservedRunningTime="2026-07-15 20:10:16.8856031 +0000 UTC m=+693.595169352" Jul 15 20:10:24.693445 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:24.693401 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:24.693874 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:24.693470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:24.706058 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:24.706030 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:24.909254 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:24.909222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq" Jul 15 20:10:27.161311 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:27.161276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:28.025376 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:28.025288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:39.590749 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:39.590662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:48.027184 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:48.027148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:53.440978 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:53.440944 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:10:53.589364 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:10:53.589311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:02.539272 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.539233 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks"] Jul 15 20:11:02.543707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.543683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.546333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.546305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 20:11:02.553120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.553094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks"] Jul 15 20:11:02.587306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnqw\" (UniqueName: \"kubernetes.io/projected/4d4459b6-3662-4341-875b-ed572200e9a2-kube-api-access-2jnqw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.587486 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.587486 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4459b6-3662-4341-875b-ed572200e9a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.587486 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.587618 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.587656 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.587612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.636141 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.636103 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:02.688822 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.688822 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4459b6-3662-4341-875b-ed572200e9a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.688950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnqw\" (UniqueName: \"kubernetes.io/projected/4d4459b6-3662-4341-875b-ed572200e9a2-kube-api-access-2jnqw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689459 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.689423 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.689504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.689726 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.689702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.691220 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.691191 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4459b6-3662-4341-875b-ed572200e9a2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.691537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.691518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4459b6-3662-4341-875b-ed572200e9a2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.696943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.696913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnqw\" (UniqueName: \"kubernetes.io/projected/4d4459b6-3662-4341-875b-ed572200e9a2-kube-api-access-2jnqw\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks\" (UID: \"4d4459b6-3662-4341-875b-ed572200e9a2\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.853867 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.853821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:02.994491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:02.994464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks"] Jul 15 20:11:02.997555 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:11:02.997530 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4459b6_3662_4341_875b_ed572200e9a2.slice/crio-4dcc71d30c059a78bbf2ba8ae7461ccb988fd7565decce090c2cb9ad1b9d9c46 WatchSource:0}: Error finding container 4dcc71d30c059a78bbf2ba8ae7461ccb988fd7565decce090c2cb9ad1b9d9c46: Status 404 returned error can't find the container with id 4dcc71d30c059a78bbf2ba8ae7461ccb988fd7565decce090c2cb9ad1b9d9c46 Jul 15 20:11:03.077089 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:03.077049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" event={"ID":"4d4459b6-3662-4341-875b-ed572200e9a2","Type":"ContainerStarted","Data":"4dcc71d30c059a78bbf2ba8ae7461ccb988fd7565decce090c2cb9ad1b9d9c46"} Jul 15 20:11:04.082776 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:04.082741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" event={"ID":"4d4459b6-3662-4341-875b-ed572200e9a2","Type":"ContainerStarted","Data":"3c1ec2e5fc46a8491a35905bd73a8e6b1483ee80f4275babebdc360f6a75c3b1"} Jul 15 20:11:04.104027 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:04.103936 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" podStartSLOduration=1.914994617 podStartE2EDuration="2.103915235s" totalImagesPullingTime="188.920618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:11:04.101784041 +0000 UTC m=+740.811350288" watchObservedRunningTime="2026-07-15 20:11:04.103915235 +0000 UTC m=+740.813481487" Jul 15 20:11:12.854740 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:12.854691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:12.855287 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:12.854760 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:12.867710 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:12.867679 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:13.137351 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:13.137250 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks" Jul 15 20:11:25.874675 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:25.874641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:56.844181 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:56.844142 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:11:56.947674 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:11:56.947639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:17.504794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.504704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8c84d6fd-vkc4x"] Jul 15 20:12:17.508109 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.508083 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.517689 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.517657 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8c84d6fd-vkc4x"] Jul 15 20:12:17.651959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.651910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zts2c\" (UniqueName: \"kubernetes.io/projected/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-kube-api-access-zts2c\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.651959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.651962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-tls-cert\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.752506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.752464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-tls-cert\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.752683 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.752557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zts2c\" (UniqueName: \"kubernetes.io/projected/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-kube-api-access-zts2c\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.755050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.754943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-tls-cert\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.764892 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.764862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zts2c\" (UniqueName: \"kubernetes.io/projected/2d5f6214-b679-4d39-8b6c-1c69c8e544cc-kube-api-access-zts2c\") pod \"authorino-8c84d6fd-vkc4x\" (UID: \"2d5f6214-b679-4d39-8b6c-1c69c8e544cc\") " pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.817894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.817858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8c84d6fd-vkc4x" Jul 15 20:12:17.974916 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.974871 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8c84d6fd-vkc4x"] Jul 15 20:12:17.976215 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:12:17.976188 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d5f6214_b679_4d39_8b6c_1c69c8e544cc.slice/crio-ea645059af451f69c6d4b4d73988485dcaf0bd240066e99fbb11dcd355b7d52d WatchSource:0}: Error finding container ea645059af451f69c6d4b4d73988485dcaf0bd240066e99fbb11dcd355b7d52d: Status 404 returned error can't find the container with id ea645059af451f69c6d4b4d73988485dcaf0bd240066e99fbb11dcd355b7d52d Jul 15 20:12:17.981601 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:17.981574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:18.082091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:18.082058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:18.433885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:18.433794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8c84d6fd-vkc4x" event={"ID":"2d5f6214-b679-4d39-8b6c-1c69c8e544cc","Type":"ContainerStarted","Data":"ea645059af451f69c6d4b4d73988485dcaf0bd240066e99fbb11dcd355b7d52d"} Jul 15 20:12:19.206732 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.206693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:19.290079 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.290042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:12:19.441826 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.441788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8c84d6fd-vkc4x" event={"ID":"2d5f6214-b679-4d39-8b6c-1c69c8e544cc","Type":"ContainerStarted","Data":"f127768b5a41788a0dfac6c44c2ce0fedb3d705a9e9470bcc3d71c3cb669a01a"} Jul 15 20:12:19.474759 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.474645 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8c84d6fd-vkc4x" podStartSLOduration=1.9665155570000001 podStartE2EDuration="2.474619965s" totalImagesPullingTime="508.104408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:12:19.472465902 +0000 UTC m=+816.182032156" watchObservedRunningTime="2026-07-15 20:12:19.474619965 +0000 UTC m=+816.184186218" Jul 15 20:12:19.522034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.521966 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:12:19.522229 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.522191 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8467fbf95-cx97c" podUID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" containerName="authorino" containerID="cri-o://1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95" gracePeriod=30 Jul 15 20:12:19.800383 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.800354 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:12:19.873386 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.873349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert\") pod \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " Jul 15 20:12:19.873572 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.873491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdmjd\" (UniqueName: \"kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd\") pod \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\" (UID: \"75811df7-fdd5-4d21-bf36-97c8d12bf5ed\") " Jul 15 20:12:19.875548 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.875513 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd" (OuterVolumeSpecName: "kube-api-access-sdmjd") pod "75811df7-fdd5-4d21-bf36-97c8d12bf5ed" (UID: "75811df7-fdd5-4d21-bf36-97c8d12bf5ed"). InnerVolumeSpecName "kube-api-access-sdmjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:12:19.884779 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.884742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "75811df7-fdd5-4d21-bf36-97c8d12bf5ed" (UID: "75811df7-fdd5-4d21-bf36-97c8d12bf5ed"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:12:19.974647 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.974600 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmjd\" (UniqueName: \"kubernetes.io/projected/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-kube-api-access-sdmjd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:12:19.974647 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:19.974645 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/75811df7-fdd5-4d21-bf36-97c8d12bf5ed-tls-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:12:20.448540 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.448505 2576 generic.go:356] "Generic (PLEG): container finished" podID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" containerID="1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95" exitCode=0 Jul 15 20:12:20.449030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.448595 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8467fbf95-cx97c" Jul 15 20:12:20.449030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.448603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8467fbf95-cx97c" event={"ID":"75811df7-fdd5-4d21-bf36-97c8d12bf5ed","Type":"ContainerDied","Data":"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95"} Jul 15 20:12:20.449030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.448644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8467fbf95-cx97c" event={"ID":"75811df7-fdd5-4d21-bf36-97c8d12bf5ed","Type":"ContainerDied","Data":"961f7933f07e062851b856f19b81342caaa1ebbb5840d894bcc26fe9033832f1"} Jul 15 20:12:20.449030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.448662 2576 scope.go:117] "RemoveContainer" containerID="1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95" Jul 15 20:12:20.458267 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.458246 2576 scope.go:117] "RemoveContainer" containerID="1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95" Jul 15 20:12:20.458617 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:12:20.458595 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95\": container with ID starting with 1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95 not found: ID does not exist" containerID="1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95" Jul 15 20:12:20.458670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.458627 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95"} err="failed to get container status \"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95\": rpc error: code = NotFound desc = could not find container \"1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95\": container with ID starting with 1788a8d27488b4729c4fe1f7950b90e63da92c7e8c2d5ddcfb0da10ab0ff2b95 not found: ID does not exist" Jul 15 20:12:20.477494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.477462 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:12:20.483297 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:20.483267 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8467fbf95-cx97c"] Jul 15 20:12:21.848001 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:21.847969 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" path="/var/lib/kubelet/pods/75811df7-fdd5-4d21-bf36-97c8d12bf5ed/volumes" Jul 15 20:12:24.345261 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:12:24.345227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:09.478591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:09.478545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:09.551228 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:09.551191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:30.516190 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:30.516153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:31.770681 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:31.770648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:13:43.723594 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:43.723555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:13:43.724246 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:13:43.724226 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:14:17.996344 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:17.996311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:18.099036 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:18.098964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:18.238532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:18.238496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:24.303153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:24.303117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:24.903245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:24.903207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:25.036968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:25.036931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:30.497880 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:30.497844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:30.708766 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:30.708723 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:14:59.236788 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:14:59.236746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:15:17.031881 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:15:17.031798 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:15:20.042876 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:15:20.042842 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:16:48.953370 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:16:48.953329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:18:34.228955 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:34.228870 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:18:34.229490 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:34.229156 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" podUID="8084e85f-2eb8-48ef-8725-c438a577b6c6" containerName="manager" containerID="cri-o://c97e6732b25edccaea1e239a22969b586af162999ec3afacf7fe0d216e0186f4" gracePeriod=10 Jul 15 20:18:38.293578 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.293484 2576 generic.go:356] "Generic (PLEG): container finished" podID="8084e85f-2eb8-48ef-8725-c438a577b6c6" containerID="c97e6732b25edccaea1e239a22969b586af162999ec3afacf7fe0d216e0186f4" exitCode=0 Jul 15 20:18:38.293578 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.293563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" event={"ID":"8084e85f-2eb8-48ef-8725-c438a577b6c6","Type":"ContainerDied","Data":"c97e6732b25edccaea1e239a22969b586af162999ec3afacf7fe0d216e0186f4"} Jul 15 20:18:38.689344 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.689310 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:18:38.815971 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.815866 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume\") pod \"8084e85f-2eb8-48ef-8725-c438a577b6c6\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " Jul 15 20:18:38.815971 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.815966 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45bt6\" (UniqueName: \"kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6\") pod \"8084e85f-2eb8-48ef-8725-c438a577b6c6\" (UID: \"8084e85f-2eb8-48ef-8725-c438a577b6c6\") " Jul 15 20:18:38.816314 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.816290 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8084e85f-2eb8-48ef-8725-c438a577b6c6" (UID: "8084e85f-2eb8-48ef-8725-c438a577b6c6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:18:38.818140 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.818115 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6" (OuterVolumeSpecName: "kube-api-access-45bt6") pod "8084e85f-2eb8-48ef-8725-c438a577b6c6" (UID: "8084e85f-2eb8-48ef-8725-c438a577b6c6"). InnerVolumeSpecName "kube-api-access-45bt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:18:38.917390 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.917345 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45bt6\" (UniqueName: \"kubernetes.io/projected/8084e85f-2eb8-48ef-8725-c438a577b6c6-kube-api-access-45bt6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:18:38.917390 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:38.917374 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8084e85f-2eb8-48ef-8725-c438a577b6c6-extensions-socket-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:18:39.299672 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.299643 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" Jul 15 20:18:39.299672 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.299658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48" event={"ID":"8084e85f-2eb8-48ef-8725-c438a577b6c6","Type":"ContainerDied","Data":"c6f964b63bcfd0ad2f6a194c6c3e7587f8a64af854c56139c8f1ebb06f6be052"} Jul 15 20:18:39.300171 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.299722 2576 scope.go:117] "RemoveContainer" containerID="c97e6732b25edccaea1e239a22969b586af162999ec3afacf7fe0d216e0186f4" Jul 15 20:18:39.322649 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.322603 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:18:39.326050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.326003 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6lq48"] Jul 15 20:18:39.848629 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:39.848593 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8084e85f-2eb8-48ef-8725-c438a577b6c6" path="/var/lib/kubelet/pods/8084e85f-2eb8-48ef-8725-c438a577b6c6/volumes" Jul 15 20:18:43.750753 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:43.750721 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:18:43.752247 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:18:43.752228 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:19:18.344573 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:18.344530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:18.456386 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:18.456348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:19:35.412408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412314 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd"] Jul 15 20:19:35.412893 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412817 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8084e85f-2eb8-48ef-8725-c438a577b6c6" containerName="manager" Jul 15 20:19:35.412893 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412837 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8084e85f-2eb8-48ef-8725-c438a577b6c6" containerName="manager" Jul 15 20:19:35.412893 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412859 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" containerName="authorino" Jul 15 20:19:35.412893 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412869 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" containerName="authorino" Jul 15 20:19:35.413151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412954 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75811df7-fdd5-4d21-bf36-97c8d12bf5ed" containerName="authorino" Jul 15 20:19:35.413151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.412965 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8084e85f-2eb8-48ef-8725-c438a577b6c6" containerName="manager" Jul 15 20:19:35.415916 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.415893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.420889 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.420866 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-db22h\"" Jul 15 20:19:35.427204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.427173 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd"] Jul 15 20:19:35.514753 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.514716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nlb\" (UniqueName: \"kubernetes.io/projected/e1856216-e281-406e-837f-1be7b670af2a-kube-api-access-84nlb\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.514917 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.514768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1856216-e281-406e-837f-1be7b670af2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.616217 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.616175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1856216-e281-406e-837f-1be7b670af2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.616425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.616271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nlb\" (UniqueName: \"kubernetes.io/projected/e1856216-e281-406e-837f-1be7b670af2a-kube-api-access-84nlb\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.616635 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.616607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1856216-e281-406e-837f-1be7b670af2a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.625831 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.625801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nlb\" (UniqueName: \"kubernetes.io/projected/e1856216-e281-406e-837f-1be7b670af2a-kube-api-access-84nlb\") pod \"kuadrant-operator-controller-manager-86878c769f-dbjcd\" (UID: \"e1856216-e281-406e-837f-1be7b670af2a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.726211 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.726098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:35.873489 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:35.873463 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd"] Jul 15 20:19:35.876063 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:19:35.876030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1856216_e281_406e_837f_1be7b670af2a.slice/crio-9b141d8686578902dbc56633ef4c2f5e6c620f17f84e9309587fbcafb7f435b4 WatchSource:0}: Error finding container 9b141d8686578902dbc56633ef4c2f5e6c620f17f84e9309587fbcafb7f435b4: Status 404 returned error can't find the container with id 9b141d8686578902dbc56633ef4c2f5e6c620f17f84e9309587fbcafb7f435b4 Jul 15 20:19:36.571365 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:36.571328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" event={"ID":"e1856216-e281-406e-837f-1be7b670af2a","Type":"ContainerStarted","Data":"a600c168b833eb206d4af8b9d71f1f173c3c76bb1eb105be62075873a7917a44"} Jul 15 20:19:36.571748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:36.571389 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:19:36.571748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:36.571406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" event={"ID":"e1856216-e281-406e-837f-1be7b670af2a","Type":"ContainerStarted","Data":"9b141d8686578902dbc56633ef4c2f5e6c620f17f84e9309587fbcafb7f435b4"} Jul 15 20:19:36.592283 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:36.592230 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" podStartSLOduration=1.592212209 podStartE2EDuration="1.592212209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:19:36.590141949 +0000 UTC m=+1253.299708201" watchObservedRunningTime="2026-07-15 20:19:36.592212209 +0000 UTC m=+1253.301778522" Jul 15 20:19:47.579457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:19:47.579424 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dbjcd" Jul 15 20:23:03.811322 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:23:03.811285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:23:43.778054 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:23:43.777999 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:23:43.779930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:23:43.779909 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:24:56.060232 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:24:56.060201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:25.528263 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:25.528223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:35.133373 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:35.133281 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:43.410884 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:43.410845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:48.557436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:48.557399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:51.699028 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:51.698983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:51.791290 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:51.791222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:52.226862 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:52.226819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.196916 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.196875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.310366 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.310330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.427641 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.427606 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:25:58.431245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.431220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.435208 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.435178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-609ae16c-openshift-default-dockercfg-rzqkr\"" Jul 15 20:25:58.461807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.461726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:25:58.486679 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.486643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.582155 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5r4\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.582591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.582511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.584266 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.584239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:58.683306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5r4\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683510 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683828 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683828 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.683941 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.683853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.684382 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.684181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.684382 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.684218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.684565 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.684404 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.684832 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.684811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.686047 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.686023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.686626 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.686600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.695430 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.695393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5r4\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.696002 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.695982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token\") pod \"e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.742335 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.742234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:58.910528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.910501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:25:58.912551 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:25:58.912522 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb350259_457a_4630_9f4e_7c5546acf45b.slice/crio-6d26e22c1cc79eaf2a5310925aac5a96689e90ed23a61d419d665948af90d65b WatchSource:0}: Error finding container 6d26e22c1cc79eaf2a5310925aac5a96689e90ed23a61d419d665948af90d65b: Status 404 returned error can't find the container with id 6d26e22c1cc79eaf2a5310925aac5a96689e90ed23a61d419d665948af90d65b Jul 15 20:25:58.914667 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.914622 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:25:58.914725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.914707 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:25:58.914761 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:58.914752 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:25:59.443336 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.443293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerStarted","Data":"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03"} Jul 15 20:25:59.443708 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.443390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerStarted","Data":"6d26e22c1cc79eaf2a5310925aac5a96689e90ed23a61d419d665948af90d65b"} Jul 15 20:25:59.489231 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.489170 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podStartSLOduration=1.489150535 podStartE2EDuration="1.489150535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:25:59.48817171 +0000 UTC m=+1636.197737962" watchObservedRunningTime="2026-07-15 20:25:59.489150535 +0000 UTC m=+1636.198716786" Jul 15 20:25:59.741812 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.741702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:25:59.742741 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.742720 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:25:59.747913 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:25:59.747881 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:00.051662 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:00.051569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:00.361945 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:00.361913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:00.449334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:00.449293 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:00.450307 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:00.450276 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:00.583529 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:00.583493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:01.165908 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:01.165874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:01.358221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:01.358191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:02.812808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:02.812768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:26:11.509626 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.509594 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb350259-457a-4630-9f4e-7c5546acf45b" containerID="7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03" exitCode=0 Jul 15 20:26:11.510079 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.509661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerDied","Data":"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03"} Jul 15 20:26:11.510130 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.510082 2576 scope.go:117] "RemoveContainer" containerID="7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03" Jul 15 20:26:11.511267 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.511236 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:26:11.511352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.511325 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:26:11.511391 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:11.511380 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:26:12.526455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:12.526412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerStarted","Data":"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda"} Jul 15 20:26:12.743317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:12.743273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:12.744610 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:12.744579 2576 patch_prober.go:28] interesting pod/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 20:26:12.744728 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:12.744636 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 20:26:13.743433 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:13.743392 2576 patch_prober.go:28] interesting pod/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 20:26:13.743837 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:13.743461 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 20:26:14.743661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:14.743626 2576 patch_prober.go:28] interesting pod/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 20:26:14.744118 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:14.743714 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 20:26:15.449668 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:15.449631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:15.743044 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:15.742928 2576 patch_prober.go:28] interesting pod/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 20:26:15.743044 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:15.742987 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 20:26:16.747957 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:16.747919 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:26:16.749193 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:26:16.749174 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:27:15.205666 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:15.205620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:19.041133 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:19.041087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:19.118137 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:19.118102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:19.820945 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:19.820907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:26.169383 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:26.169343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:26.244102 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:26.244065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:30.024233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.024197 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:27:30.024737 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.024437 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" containerID="cri-o://6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda" gracePeriod=30 Jul 15 20:27:30.061647 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.061610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:30.502874 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.502839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:30.661019 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.660973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:30.792653 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.792566 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:27:30.795961 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.795939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.800835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.800811 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-be7332c6-gw-openshift-default-dockercfg-4v78z\"" Jul 15 20:27:30.828835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.828800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:27:30.911136 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfv9k\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911331 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911327 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.911729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.911439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:30.967416 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:30.967373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:31.012625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.012625 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.012835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.012882 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.012923 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.012962 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.012925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfv9k\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013187 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013296 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013394 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013357 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013394 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013508 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.013581 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.013559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.015282 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.015263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.015517 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.015498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.033527 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.033489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfv9k\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.033868 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.033664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token\") pod \"e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.106452 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.106413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:31.291726 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.291681 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:27:31.299116 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:27:31.299076 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245ecef4_4704_4f56_b2b3_378bafc825d4.slice/crio-c6e338a6d3a954c443ddc14f2b8313bd29b1511c0500316aeb0119d4494f9166 WatchSource:0}: Error finding container c6e338a6d3a954c443ddc14f2b8313bd29b1511c0500316aeb0119d4494f9166: Status 404 returned error can't find the container with id c6e338a6d3a954c443ddc14f2b8313bd29b1511c0500316aeb0119d4494f9166 Jul 15 20:27:31.301280 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.301245 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:31.301380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.301315 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:31.301380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.301355 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:31.751039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.750967 2576 patch_prober.go:28] interesting pod/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:27:31.751205 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.751062 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:27:31.908337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.908301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerStarted","Data":"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267"} Jul 15 20:27:31.908524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.908361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerStarted","Data":"c6e338a6d3a954c443ddc14f2b8313bd29b1511c0500316aeb0119d4494f9166"} Jul 15 20:27:31.939704 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:31.939645 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podStartSLOduration=1.939626418 podStartE2EDuration="1.939626418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:27:31.938630742 +0000 UTC m=+1728.648197005" watchObservedRunningTime="2026-07-15 20:27:31.939626418 +0000 UTC m=+1728.649192734" Jul 15 20:27:32.091250 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.091213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:32.107534 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.107493 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:32.112884 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.112854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:32.353844 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.353750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:32.650119 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.650032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:32.913236 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.913148 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:32.914207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:32.914185 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:33.436333 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:33.436296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:33.598559 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:33.598524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:35.319601 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.319520 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:27:35.453294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453364 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453452 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms5r4\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453482 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453510 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453583 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453580 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:35.453723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket\") pod \"fb350259-457a-4630-9f4e-7c5546acf45b\" (UID: \"fb350259-457a-4630-9f4e-7c5546acf45b\") " Jul 15 20:27:35.453977 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.453832 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:35.454097 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454073 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:35.454228 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:27:35.454517 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454499 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.454683 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454671 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb350259-457a-4630-9f4e-7c5546acf45b-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.454772 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454761 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.454850 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454838 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.454930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.454870 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:35.456121 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.456094 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:27:35.456457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.456430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4" (OuterVolumeSpecName: "kube-api-access-ms5r4") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "kube-api-access-ms5r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:27:35.456457 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.456448 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:27:35.456582 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.456494 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb350259-457a-4630-9f4e-7c5546acf45b" (UID: "fb350259-457a-4630-9f4e-7c5546acf45b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:27:35.555852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.555808 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb350259-457a-4630-9f4e-7c5546acf45b-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.555852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.555842 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.555852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.555851 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5r4\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-kube-api-access-ms5r4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.555852 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.555863 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb350259-457a-4630-9f4e-7c5546acf45b-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.556186 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.555871 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb350259-457a-4630-9f4e-7c5546acf45b-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:27:35.929026 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.928976 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb350259-457a-4630-9f4e-7c5546acf45b" containerID="6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda" exitCode=0 Jul 15 20:27:35.929202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.929046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" Jul 15 20:27:35.929202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.929050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerDied","Data":"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda"} Jul 15 20:27:35.929202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.929107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l" event={"ID":"fb350259-457a-4630-9f4e-7c5546acf45b","Type":"ContainerDied","Data":"6d26e22c1cc79eaf2a5310925aac5a96689e90ed23a61d419d665948af90d65b"} Jul 15 20:27:35.929202 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.929131 2576 scope.go:117] "RemoveContainer" containerID="6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda" Jul 15 20:27:35.938228 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.938210 2576 scope.go:117] "RemoveContainer" containerID="7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03" Jul 15 20:27:35.947286 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.947266 2576 scope.go:117] "RemoveContainer" containerID="6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda" Jul 15 20:27:35.947647 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:27:35.947627 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda\": container with ID starting with 6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda not found: ID does not exist" containerID="6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda" Jul 15 20:27:35.947699 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.947660 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda"} err="failed to get container status \"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda\": rpc error: code = NotFound desc = could not find container \"6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda\": container with ID starting with 6fd998dee48aea1fabb53a1f94d4e99bb8f9a01467836f7b4e7d0651b9330fda not found: ID does not exist" Jul 15 20:27:35.947699 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.947683 2576 scope.go:117] "RemoveContainer" containerID="7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03" Jul 15 20:27:35.947945 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:27:35.947928 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03\": container with ID starting with 7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03 not found: ID does not exist" containerID="7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03" Jul 15 20:27:35.947996 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.947951 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03"} err="failed to get container status \"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03\": rpc error: code = NotFound desc = could not find container \"7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03\": container with ID starting with 7388b8b74d34908cea6be108b141719a313bf91218045a0258748b20a1791b03 not found: ID does not exist" Jul 15 20:27:35.954600 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.954563 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:27:35.960107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:35.960078 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-609ae16c-openshift-default-567c7b6cf7-8sp8l"] Jul 15 20:27:36.313582 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:36.313478 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:36.400599 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:36.400390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:37.848373 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:37.848336 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" path="/var/lib/kubelet/pods/fb350259-457a-4630-9f4e-7c5546acf45b/volumes" Jul 15 20:27:41.802989 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:41.802956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:41.991436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:41.991398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:42.967309 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.967275 2576 generic.go:356] "Generic (PLEG): container finished" podID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerID="5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267" exitCode=0 Jul 15 20:27:42.967747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.967349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerDied","Data":"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267"} Jul 15 20:27:42.967747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.967723 2576 scope.go:117] "RemoveContainer" containerID="5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267" Jul 15 20:27:42.968805 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.968776 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:42.968880 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.968847 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:42.968918 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:42.968883 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:27:43.974044 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:43.973989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerStarted","Data":"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad"} Jul 15 20:27:44.110063 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:44.107212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:44.112351 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:44.111141 2576 patch_prober.go:28] interesting pod/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 20:27:44.112351 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:44.111222 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 20:27:45.107097 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:45.107050 2576 patch_prober.go:28] interesting pod/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 20:27:45.107612 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:45.107121 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 20:27:45.219321 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:45.219283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:27:46.107035 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:46.106973 2576 patch_prober.go:28] interesting pod/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 20:27:46.107266 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:46.107069 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 20:27:47.106992 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:47.106943 2576 patch_prober.go:28] interesting pod/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 20:27:47.107239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:47.107047 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 20:27:47.913306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:47.913269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:48.111208 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:48.111178 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:27:48.112292 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:27:48.112273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:28:36.689620 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:36.689529 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:28:36.690139 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:36.689801 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" containerID="cri-o://b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad" gracePeriod=30 Jul 15 20:28:36.724330 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:36.724295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:38.681960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:38.681917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:41.954281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:41.954205 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:28:42.039705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.039705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.039959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.039959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039760 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.039959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.039959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.039824 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.040197 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040132 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:28:42.040197 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040148 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfv9k\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.040197 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.040361 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040219 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket\") pod \"245ecef4-4704-4f56-b2b3-378bafc825d4\" (UID: \"245ecef4-4704-4f56-b2b3-378bafc825d4\") " Jul 15 20:28:42.040361 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040234 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:28:42.040505 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040485 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:28:42.040579 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040565 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.040637 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040586 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245ecef4-4704-4f56-b2b3-378bafc825d4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.040637 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040600 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.040637 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040564 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:28:42.040786 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.040698 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:28:42.042169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.042144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:28:42.042518 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.042486 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:28:42.042630 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.042532 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k" (OuterVolumeSpecName: "kube-api-access-nfv9k") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "kube-api-access-nfv9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:28:42.042630 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.042571 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "245ecef4-4704-4f56-b2b3-378bafc825d4" (UID: "245ecef4-4704-4f56-b2b3-378bafc825d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:28:42.142093 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142053 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.142093 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142087 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.142093 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142096 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.142334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142118 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.142334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142127 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfv9k\" (UniqueName: \"kubernetes.io/projected/245ecef4-4704-4f56-b2b3-378bafc825d4-kube-api-access-nfv9k\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.142334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.142139 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245ecef4-4704-4f56-b2b3-378bafc825d4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:28:42.255936 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.255835 2576 generic.go:356] "Generic (PLEG): container finished" podID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerID="b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad" exitCode=0 Jul 15 20:28:42.255936 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.255880 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" Jul 15 20:28:42.255936 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.255910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerDied","Data":"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad"} Jul 15 20:28:42.256163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.255969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq" event={"ID":"245ecef4-4704-4f56-b2b3-378bafc825d4","Type":"ContainerDied","Data":"c6e338a6d3a954c443ddc14f2b8313bd29b1511c0500316aeb0119d4494f9166"} Jul 15 20:28:42.256163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.255993 2576 scope.go:117] "RemoveContainer" containerID="b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad" Jul 15 20:28:42.265272 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.265252 2576 scope.go:117] "RemoveContainer" containerID="5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267" Jul 15 20:28:42.273602 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.273579 2576 scope.go:117] "RemoveContainer" containerID="b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad" Jul 15 20:28:42.273923 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:28:42.273897 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad\": container with ID starting with b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad not found: ID does not exist" containerID="b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad" Jul 15 20:28:42.274000 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.273937 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad"} err="failed to get container status \"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad\": rpc error: code = NotFound desc = could not find container \"b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad\": container with ID starting with b4d7578c8e1d958dfc5ae9125515a028a1e56990053994e6a0f84a47766180ad not found: ID does not exist" Jul 15 20:28:42.274000 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.273964 2576 scope.go:117] "RemoveContainer" containerID="5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267" Jul 15 20:28:42.274265 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:28:42.274246 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267\": container with ID starting with 5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267 not found: ID does not exist" containerID="5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267" Jul 15 20:28:42.274306 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.274273 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267"} err="failed to get container status \"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267\": rpc error: code = NotFound desc = could not find container \"5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267\": container with ID starting with 5c21853ab1ac50f0ee66789537a4b31d282adb0336446b36d950e61631779267 not found: ID does not exist" Jul 15 20:28:42.278003 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.277978 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:28:42.281630 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.281603 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-be7332c6-gw-openshift-default-86dd4b4559-5zgxq"] Jul 15 20:28:42.809501 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.809462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:42.871184 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.871151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:42.996979 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.996949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:28:42.997427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997372 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:28:42.997427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997385 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:28:42.997427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997397 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:28:42.997427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997405 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:28:42.997427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997426 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:28:42.997591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997432 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:28:42.997591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997506 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:28:42.997591 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:42.997520 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:28:43.002136 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.002108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.004734 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.004705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3ba75ea-openshift-default-dockercfg-wp6qr\"" Jul 15 20:28:43.012864 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.012836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:28:43.028505 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.028475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:43.150648 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.150838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.150838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.150838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.150838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.150838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8hh\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.151045 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.151045 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.151045 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.150968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.251799 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.251759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.251799 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.251806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.251849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.251881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8hh\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252376 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252708 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.252708 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.252708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.254732 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.254705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.255063 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.255040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.262057 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.262032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.262233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.262200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8hh\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh\") pod \"e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.299568 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.299530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:43.314488 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.314451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:43.402894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.402802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:43.468278 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.468253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:28:43.470470 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:28:43.470435 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b51b429_48e4_4e6f_8b3d_46a55e9d52e4.slice/crio-406942282be3f79af697d694bf62f1b6b509aac08b87e237a38cbcc07c64c022 WatchSource:0}: Error finding container 406942282be3f79af697d694bf62f1b6b509aac08b87e237a38cbcc07c64c022: Status 404 returned error can't find the container with id 406942282be3f79af697d694bf62f1b6b509aac08b87e237a38cbcc07c64c022 Jul 15 20:28:43.473198 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.473155 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:43.473305 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.473253 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:43.473305 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.473299 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:43.806814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.806732 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:28:43.810556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.810531 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:28:43.848364 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:43.848325 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" path="/var/lib/kubelet/pods/245ecef4-4704-4f56-b2b3-378bafc825d4/volumes" Jul 15 20:28:44.161259 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.161224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:44.254915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.254883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:44.268381 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.268342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerStarted","Data":"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83"} Jul 15 20:28:44.268550 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.268400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerStarted","Data":"406942282be3f79af697d694bf62f1b6b509aac08b87e237a38cbcc07c64c022"} Jul 15 20:28:44.298431 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.298093 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podStartSLOduration=2.298073028 podStartE2EDuration="2.298073028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:28:44.29757447 +0000 UTC m=+1801.007140723" watchObservedRunningTime="2026-07-15 20:28:44.298073028 +0000 UTC m=+1801.007639279" Jul 15 20:28:44.314723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.314689 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:44.319742 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.319715 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:44.348344 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.348313 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:44.859818 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.859784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:44.963793 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:44.963743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:45.060048 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:45.059997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:45.279621 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:45.279521 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:45.280454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:45.280431 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:47.495211 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:47.495175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:53.174160 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:53.174127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:53.273673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:53.273641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:28:56.341974 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.341937 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerID="3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83" exitCode=0 Jul 15 20:28:56.342512 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.342032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerDied","Data":"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83"} Jul 15 20:28:56.342583 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.342531 2576 scope.go:117] "RemoveContainer" containerID="3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83" Jul 15 20:28:56.343614 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.343581 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:56.343707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.343657 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:56.343707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:56.343688 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:28:57.348594 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:57.348557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerStarted","Data":"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f"} Jul 15 20:28:58.315273 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:58.315233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:28:58.316536 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:58.316505 2576 patch_prober.go:28] interesting pod/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:28:58.316673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:58.316573 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 15 20:28:59.315485 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:59.315441 2576 patch_prober.go:28] interesting pod/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:28:59.315984 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:28:59.315527 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 15 20:29:00.279985 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:00.279935 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:29:00.315102 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:00.315064 2576 patch_prober.go:28] interesting pod/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:29:00.315290 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:00.315131 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 15 20:29:01.315521 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:01.315467 2576 patch_prober.go:28] interesting pod/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 15 20:29:01.316082 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:01.315546 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 15 20:29:02.318869 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:02.318832 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:29:02.319880 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:02.319861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:29:05.619161 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:05.619112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:29:05.727284 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:05.727233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:29:49.801469 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:49.801431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:29:54.424446 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:29:54.424406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:00.410760 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:00.410727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:00.528335 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:00.528298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:05.457796 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.457703 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:30:05.458250 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.457987 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" containerID="cri-o://bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f" gracePeriod=30 Jul 15 20:30:05.485481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.485444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:05.767569 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.767466 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:30:05.767939 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.767920 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:30:05.768061 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.767942 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:30:05.768123 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.768073 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="245ecef4-4704-4f56-b2b3-378bafc825d4" containerName="istio-proxy" Jul 15 20:30:05.768123 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.768091 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb350259-457a-4630-9f4e-7c5546acf45b" containerName="istio-proxy" Jul 15 20:30:05.771184 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.771157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.774199 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.774176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6dc6ee57-openshift-default-dockercfg-j28sr\"" Jul 15 20:30:05.787163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.787127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:30:05.802509 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.802472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:05.917121 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917422 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917422 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917561 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:05.917650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:05.917633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7x57\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018540 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018540 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018540 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018784 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7x57\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018784 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018784 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018940 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018940 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.018940 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.018926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.019142 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.019121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.019217 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.019195 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.019318 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.019300 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.019377 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.019354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.019794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.019769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.021060 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.021034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.021326 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.021303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.027076 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.027049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7x57\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.027195 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.027049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token\") pod \"e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.082098 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.082046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:06.238158 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.238123 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:30:06.239800 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:30:06.239775 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42306bcd_51cd_4405_9c54_fe4418186cad.slice/crio-84a93ee75766392d19bae9965420a0e5385718da30ac13d70feb456c4bd5bb49 WatchSource:0}: Error finding container 84a93ee75766392d19bae9965420a0e5385718da30ac13d70feb456c4bd5bb49: Status 404 returned error can't find the container with id 84a93ee75766392d19bae9965420a0e5385718da30ac13d70feb456c4bd5bb49 Jul 15 20:30:06.242458 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.242425 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:06.242539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.242508 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:06.242581 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.242538 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:06.699030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.698975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerStarted","Data":"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040"} Jul 15 20:30:06.699511 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.699050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerStarted","Data":"84a93ee75766392d19bae9965420a0e5385718da30ac13d70feb456c4bd5bb49"} Jul 15 20:30:06.722802 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.722742 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podStartSLOduration=1.722721928 podStartE2EDuration="1.722721928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:30:06.72027027 +0000 UTC m=+1883.429836525" watchObservedRunningTime="2026-07-15 20:30:06.722721928 +0000 UTC m=+1883.432288180" Jul 15 20:30:06.970456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:06.970345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:07.068887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.068835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:07.082512 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.082471 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:07.089134 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.089101 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:07.152913 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.152878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:07.468892 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.468851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:07.659070 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.659037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:07.705172 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.705142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:07.706327 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:07.706307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:08.095372 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:08.095335 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:10.719505 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.719428 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:30:10.720502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.720478 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerID="bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f" exitCode=0 Jul 15 20:30:10.720604 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.720550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerDied","Data":"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f"} Jul 15 20:30:10.720673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.720617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" event={"ID":"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4","Type":"ContainerDied","Data":"406942282be3f79af697d694bf62f1b6b509aac08b87e237a38cbcc07c64c022"} Jul 15 20:30:10.720673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.720653 2576 scope.go:117] "RemoveContainer" containerID="bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f" Jul 15 20:30:10.729557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.729532 2576 scope.go:117] "RemoveContainer" containerID="3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83" Jul 15 20:30:10.739242 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.739217 2576 scope.go:117] "RemoveContainer" containerID="bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f" Jul 15 20:30:10.739601 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:30:10.739577 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f\": container with ID starting with bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f not found: ID does not exist" containerID="bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f" Jul 15 20:30:10.739695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.739615 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f"} err="failed to get container status \"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f\": rpc error: code = NotFound desc = could not find container \"bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f\": container with ID starting with bae468ba6bd6add5b03886f7a4883373669e555aaaa26c6743740df1fc09454f not found: ID does not exist" Jul 15 20:30:10.739695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.739641 2576 scope.go:117] "RemoveContainer" containerID="3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83" Jul 15 20:30:10.739947 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:30:10.739929 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83\": container with ID starting with 3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83 not found: ID does not exist" containerID="3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83" Jul 15 20:30:10.740022 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.739957 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83"} err="failed to get container status \"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83\": rpc error: code = NotFound desc = could not find container \"3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83\": container with ID starting with 3238399f75aafe7df83eaff2b5b2bd3e3ce4d44ee312547618a4571cccfbae83 not found: ID does not exist" Jul 15 20:30:10.867885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.867820 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.867885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.867894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868190 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.867924 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868190 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.867969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868190 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.867998 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr8hh\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868190 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868200 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:30:10.868395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data\") pod \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\" (UID: \"7b51b429-48e4-4e6f-8b3d-46a55e9d52e4\") " Jul 15 20:30:10.868395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868378 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:30:10.868641 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868407 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:30:10.868641 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868592 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:30:10.868743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868659 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.868743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868678 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.868743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.868692 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.869124 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.869101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:30:10.870454 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.870422 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:30:10.870564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.870475 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh" (OuterVolumeSpecName: "kube-api-access-xr8hh") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "kube-api-access-xr8hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:30:10.870705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.870685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:30:10.870765 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.870751 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" (UID: "7b51b429-48e4-4e6f-8b3d-46a55e9d52e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970104 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970135 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970144 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970152 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970163 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8hh\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-kube-api-access-xr8hh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:10.970194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:10.970172 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:30:11.726954 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:11.726914 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z" Jul 15 20:30:11.748748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:11.748698 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:30:11.750458 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:11.750434 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3ba75ea-openshift-default-86ddb9c999-msf4z"] Jul 15 20:30:11.848665 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:11.848629 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" path="/var/lib/kubelet/pods/7b51b429-48e4-4e6f-8b3d-46a55e9d52e4/volumes" Jul 15 20:30:12.253549 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:12.253515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:12.324700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:12.324646 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:16.187621 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:16.187581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:30:23.279239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.279206 2576 patch_prober.go:28] interesting pod/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:42110->10.134.0.46:15021: read: connection reset by peer" start-of-body= Jul 15 20:30:23.279583 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.279315 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": read tcp 10.134.0.2:42110->10.134.0.46:15021: read: connection reset by peer" Jul 15 20:30:23.788959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.788923 2576 generic.go:356] "Generic (PLEG): container finished" podID="42306bcd-51cd-4405-9c54-fe4418186cad" containerID="96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040" exitCode=0 Jul 15 20:30:23.789183 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.788995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerDied","Data":"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040"} Jul 15 20:30:23.789494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.789472 2576 scope.go:117] "RemoveContainer" containerID="96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040" Jul 15 20:30:23.790678 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.790640 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:23.790769 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.790739 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:23.790814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:23.790785 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:30:24.799894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:24.799849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerStarted","Data":"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2"} Jul 15 20:30:25.082596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:25.082555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:25.082912 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:25.082886 2576 patch_prober.go:28] interesting pod/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 20:30:25.082996 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:25.082948 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 20:30:26.082491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:26.082454 2576 patch_prober.go:28] interesting pod/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 20:30:26.082888 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:26.082513 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 20:30:27.082563 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:27.082521 2576 patch_prober.go:28] interesting pod/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 20:30:27.082960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:27.082583 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 20:30:28.082521 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:28.082473 2576 patch_prober.go:28] interesting pod/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 20:30:28.082707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:28.082561 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 20:30:29.086738 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:29.086706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:29.087201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:29.086980 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:30:29.087661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:30:29.087645 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:31:22.034925 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:22.034859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:26.551753 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:26.551712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:32.582105 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:32.582063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:32.682448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:32.682414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:36.721687 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:36.721585 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:31:36.722204 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:36.721870 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" containerID="cri-o://8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2" gracePeriod=30 Jul 15 20:31:36.740240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:36.740196 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:37.406738 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:37.406701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:37.510643 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:37.510606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:37.608468 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:37.608431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:39.104481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:39.104447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:41.982947 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:41.982875 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:31:42.069793 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069761 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.069930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069809 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.069930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.069930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.069930 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069896 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.070167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.069983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.070167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7x57\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.070167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.070167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070090 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token\") pod \"42306bcd-51cd-4405-9c54-fe4418186cad\" (UID: \"42306bcd-51cd-4405-9c54-fe4418186cad\") " Jul 15 20:31:42.070167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070139 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:31:42.070436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070202 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:31:42.070436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070370 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:31:42.070537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070443 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:31:42.070537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070387 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.070640 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.070542 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.071186 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.071162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data" (OuterVolumeSpecName: "istio-data") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:31:42.072397 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.072366 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token" (OuterVolumeSpecName: "istio-token") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:31:42.072594 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.072566 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:31:42.072723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.072702 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:31:42.072723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.072712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57" (OuterVolumeSpecName: "kube-api-access-c7x57") pod "42306bcd-51cd-4405-9c54-fe4418186cad" (UID: "42306bcd-51cd-4405-9c54-fe4418186cad"). InnerVolumeSpecName "kube-api-access-c7x57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:31:42.171988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.171951 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.171988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.171981 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.171988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.171990 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7x57\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-kube-api-access-c7x57\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.171988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.171998 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42306bcd-51cd-4405-9c54-fe4418186cad-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.172282 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.172037 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42306bcd-51cd-4405-9c54-fe4418186cad-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.172282 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.172045 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42306bcd-51cd-4405-9c54-fe4418186cad-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.172282 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.172052 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42306bcd-51cd-4405-9c54-fe4418186cad-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:31:42.185161 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.185127 2576 generic.go:356] "Generic (PLEG): container finished" podID="42306bcd-51cd-4405-9c54-fe4418186cad" containerID="8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2" exitCode=0 Jul 15 20:31:42.185337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.185194 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" Jul 15 20:31:42.185337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.185220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerDied","Data":"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2"} Jul 15 20:31:42.185337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.185279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n" event={"ID":"42306bcd-51cd-4405-9c54-fe4418186cad","Type":"ContainerDied","Data":"84a93ee75766392d19bae9965420a0e5385718da30ac13d70feb456c4bd5bb49"} Jul 15 20:31:42.185337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.185306 2576 scope.go:117] "RemoveContainer" containerID="8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2" Jul 15 20:31:42.195167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.194948 2576 scope.go:117] "RemoveContainer" containerID="96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040" Jul 15 20:31:42.204239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.204215 2576 scope.go:117] "RemoveContainer" containerID="8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2" Jul 15 20:31:42.204575 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:31:42.204550 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2\": container with ID starting with 8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2 not found: ID does not exist" containerID="8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2" Jul 15 20:31:42.204678 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.204584 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2"} err="failed to get container status \"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2\": rpc error: code = NotFound desc = could not find container \"8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2\": container with ID starting with 8704d87a4bf6ec3d6f9d2397dcd4fee1aeb2c5190337219629052829966edfb2 not found: ID does not exist" Jul 15 20:31:42.204678 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.204603 2576 scope.go:117] "RemoveContainer" containerID="96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040" Jul 15 20:31:42.204885 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:31:42.204865 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040\": container with ID starting with 96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040 not found: ID does not exist" containerID="96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040" Jul 15 20:31:42.204939 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.204893 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040"} err="failed to get container status \"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040\": rpc error: code = NotFound desc = could not find container \"96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040\": container with ID starting with 96f383a0d4996fc20c2f5a281793e1e42613d840502e6a9e9949c940e53fa040 not found: ID does not exist" Jul 15 20:31:42.209449 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.209418 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:31:42.212775 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:42.212747 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6dc6ee57-openshift-default-cfb64ff-7kt9n"] Jul 15 20:31:43.849642 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:43.849605 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" path="/var/lib/kubelet/pods/42306bcd-51cd-4405-9c54-fe4418186cad/volumes" Jul 15 20:31:45.481001 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:45.480964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:45.664962 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:45.664924 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:45.738727 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:45.738624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:45.870506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:45.870454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:45.964756 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:45.964716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:46.408410 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:46.408346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:31:53.810817 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:31:53.810782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:12.593376 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:12.593308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:18.068542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:18.068493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:18.167225 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:18.167186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:18.268794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:18.268757 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:18.367452 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:18.367414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:34.201024 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:34.200973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:39.849196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:39.849153 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:39.912067 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:39.912003 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:40.005900 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:40.005857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:40.106324 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:40.106210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:50.590775 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:50.590731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:51.444279 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:51.444241 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:57.531376 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:57.531322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:57.642598 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:57.642557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:57.770695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:57.770623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:57.843627 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:57.843596 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:58.806243 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:58.806208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:32:58.984453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:32:58.984412 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:00.663229 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:00.663188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:06.668917 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:06.668818 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:06.808840 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:06.808786 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:07.433619 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:07.433542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:13.164269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.164232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:13.268680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.268644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:13.411879 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.411828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:13.692635 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.692600 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:13.692990 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.692977 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.693091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.692991 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.693091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693000 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.693091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693021 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.693091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693033 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:33:13.693091 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693041 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:33:13.693357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693122 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:33:13.693357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693132 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.693357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.693140 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b51b429-48e4-4e6f-8b3d-46a55e9d52e4" containerName="istio-proxy" Jul 15 20:33:13.696156 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.696139 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.698762 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.698741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cea3e2bc-openshift-default-dockercfg-f6kkk\"" Jul 15 20:33:13.717797 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.717765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:13.723365 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.723336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:13.867072 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.866990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867072 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsw86\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.867312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.867272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.968814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.968710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.968814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.968751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.968814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.968774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.968905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.968956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsw86\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969458 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969458 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969281 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969536 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969482 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969536 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969512 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.969797 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.969767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.971406 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.971387 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.971618 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.971603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.977943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.977921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:13.978262 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:13.978243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsw86\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86\") pod \"e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:14.006458 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.006420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:14.162642 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.162610 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:14.164227 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:33:14.164197 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66b58de_64a7_45bd_a993_d9ad4fe64bb3.slice/crio-65986fdee729d47d157135d627a1a14029999bead1ab76cdd579e6539f8cb3bc WatchSource:0}: Error finding container 65986fdee729d47d157135d627a1a14029999bead1ab76cdd579e6539f8cb3bc: Status 404 returned error can't find the container with id 65986fdee729d47d157135d627a1a14029999bead1ab76cdd579e6539f8cb3bc Jul 15 20:33:14.166432 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.166395 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:14.166731 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.166467 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:14.166731 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.166500 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:14.342216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.342178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:14.636360 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.636323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerStarted","Data":"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384"} Jul 15 20:33:14.636575 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.636378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerStarted","Data":"65986fdee729d47d157135d627a1a14029999bead1ab76cdd579e6539f8cb3bc"} Jul 15 20:33:14.662874 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:14.662819 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podStartSLOduration=1.662800872 podStartE2EDuration="1.662800872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:33:14.661391498 +0000 UTC m=+2071.370957752" watchObservedRunningTime="2026-07-15 20:33:14.662800872 +0000 UTC m=+2071.372367124" Jul 15 20:33:15.007339 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:15.007236 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:15.012165 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:15.012135 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:15.641239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:15.641210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:15.642244 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:15.642225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:16.722864 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:16.722822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:16.911613 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:16.911574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:17.097503 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:17.097463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:17.729876 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:17.729839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:17.827844 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:17.827813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:21.166359 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:21.166329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:21.906756 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:21.906683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:22.004030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:22.003984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:22.319926 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:22.319835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:27.702960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.702920 2576 generic.go:356] "Generic (PLEG): container finished" podID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerID="2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384" exitCode=0 Jul 15 20:33:27.703398 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.702998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerDied","Data":"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384"} Jul 15 20:33:27.703443 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.703431 2576 scope.go:117] "RemoveContainer" containerID="2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384" Jul 15 20:33:27.704404 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.704376 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:27.704456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.704444 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:27.704517 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:27.704504 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:33:28.134378 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:28.134336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:28.203246 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:28.203190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:28.710350 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:28.710313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerStarted","Data":"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275"} Jul 15 20:33:29.007651 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:29.007545 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:29.008603 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:29.008567 2576 patch_prober.go:28] interesting pod/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:33:29.008801 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:29.008628 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 20:33:30.007801 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:30.007754 2576 patch_prober.go:28] interesting pod/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:33:30.008453 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:30.008420 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 20:33:30.641997 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:30.641942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:31.007700 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:31.007595 2576 patch_prober.go:28] interesting pod/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:33:31.007997 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:31.007962 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 20:33:32.007519 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:32.007463 2576 patch_prober.go:28] interesting pod/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 20:33:32.007745 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:32.007565 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 20:33:33.012960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:33.012923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:33.014087 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:33.014066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:38.957920 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:38.957886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:43.577084 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:43.577042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:43.834517 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:43.834489 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:33:43.840539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:43.840514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:33:48.925909 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:48.925874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:49.063528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:49.063486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:49.580840 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:49.580800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:49.645231 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:49.645181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:53.935595 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:53.935549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.039220 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.039161 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.271002 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.270905 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:54.271234 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.271173 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" containerID="cri-o://f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275" gracePeriod=30 Jul 15 20:33:54.282843 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.282791 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.539492 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.539401 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.646968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.646932 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.751778 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.751743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:54.895320 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:54.895284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:33:59.529207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.529179 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:59.703837 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.703782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.703837 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.703846 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.703925 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.703954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.703986 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704053 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsw86\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704127 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704153 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token\") pod \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\" (UID: \"b66b58de-64a7-45bd-a993-d9ad4fe64bb3\") " Jul 15 20:33:59.704500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:33:59.704500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:33:59.704500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704459 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.704667 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704477 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.704667 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704658 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:33:59.704779 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:33:59.704870 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.704849 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:33:59.706286 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.706252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:33:59.706406 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.706320 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:33:59.706570 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.706549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86" (OuterVolumeSpecName: "kube-api-access-lsw86") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "kube-api-access-lsw86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:33:59.706614 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.706563 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b66b58de-64a7-45bd-a993-d9ad4fe64bb3" (UID: "b66b58de-64a7-45bd-a993-d9ad4fe64bb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:33:59.805652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805599 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805645 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsw86\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-kube-api-access-lsw86\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805656 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805668 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805676 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805973 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805684 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.805973 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.805692 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b66b58de-64a7-45bd-a993-d9ad4fe64bb3-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:33:59.865853 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.865823 2576 generic.go:356] "Generic (PLEG): container finished" podID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerID="f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275" exitCode=0 Jul 15 20:33:59.866034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.865870 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" Jul 15 20:33:59.866034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.865906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerDied","Data":"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275"} Jul 15 20:33:59.866034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.865961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62" event={"ID":"b66b58de-64a7-45bd-a993-d9ad4fe64bb3","Type":"ContainerDied","Data":"65986fdee729d47d157135d627a1a14029999bead1ab76cdd579e6539f8cb3bc"} Jul 15 20:33:59.866034 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.865986 2576 scope.go:117] "RemoveContainer" containerID="f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275" Jul 15 20:33:59.875914 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.875895 2576 scope.go:117] "RemoveContainer" containerID="2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384" Jul 15 20:33:59.882823 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.882789 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:59.886081 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.886054 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cea3e2bc-openshift-default-7c784f4f87-jds62"] Jul 15 20:33:59.888294 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.888275 2576 scope.go:117] "RemoveContainer" containerID="f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275" Jul 15 20:33:59.888619 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:33:59.888597 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275\": container with ID starting with f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275 not found: ID does not exist" containerID="f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275" Jul 15 20:33:59.888661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.888630 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275"} err="failed to get container status \"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275\": rpc error: code = NotFound desc = could not find container \"f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275\": container with ID starting with f7e45a8f03b9bdbec2642b4c07c0e32767943a9931f2f5f6c122a6af66f7c275 not found: ID does not exist" Jul 15 20:33:59.888661 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.888651 2576 scope.go:117] "RemoveContainer" containerID="2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384" Jul 15 20:33:59.888929 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:33:59.888910 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384\": container with ID starting with 2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384 not found: ID does not exist" containerID="2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384" Jul 15 20:33:59.888978 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:33:59.888936 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384"} err="failed to get container status \"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384\": rpc error: code = NotFound desc = could not find container \"2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384\": container with ID starting with 2b8f30b7a6bceddd72daff92766af4fcf35f16b233836576020114c5e8b0a384 not found: ID does not exist" Jul 15 20:34:00.568527 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:00.568486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:00.629207 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:00.629179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:00.729538 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:00.729494 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:00.853248 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:00.853214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:00.994005 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:00.993970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:01.854122 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:01.854087 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" path="/var/lib/kubelet/pods/b66b58de-64a7-45bd-a993-d9ad4fe64bb3/volumes" Jul 15 20:34:06.766394 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:06.766361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:06.830295 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:06.830265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:13.737612 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:13.737565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:14.177590 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.177547 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:34:14.177987 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.177968 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.177989 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178005 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178028 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178044 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:34:14.178120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178053 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:34:14.178363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178165 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178179 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66b58de-64a7-45bd-a993-d9ad4fe64bb3" containerName="istio-proxy" Jul 15 20:34:14.178363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.178188 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="42306bcd-51cd-4405-9c54-fe4418186cad" containerName="istio-proxy" Jul 15 20:34:14.182676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.182651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.186233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.186208 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2bf8d569-openshift-default-dockercfg-df2tt\"" Jul 15 20:34:14.206792 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.206761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:14.214531 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.214494 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:34:14.231269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.231225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.231566 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.231543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vsl\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.231748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.231732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.231891 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.231876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.232122 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.232104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.232312 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.232296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.232484 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.232469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.232633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.232618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.232772 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.232758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334087 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334087 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334360 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334360 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334360 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334360 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vsl\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334573 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.334758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.334683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.335402 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.335148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.335402 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.335285 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.335402 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.335371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.337344 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.337310 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.337604 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.337579 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.343941 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.343903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.345158 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.345137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vsl\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl\") pod \"e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.470951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.470839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:14.497436 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.497402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:14.652496 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.652467 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:34:14.654922 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:34:14.654891 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894c44bb_18a3_4e11_9a77_9842be517c1e.slice/crio-53aef422a8dd0f6ee4f5cd1f0113f7de03e091a84cec390cc75ba1e1949d3a87 WatchSource:0}: Error finding container 53aef422a8dd0f6ee4f5cd1f0113f7de03e091a84cec390cc75ba1e1949d3a87: Status 404 returned error can't find the container with id 53aef422a8dd0f6ee4f5cd1f0113f7de03e091a84cec390cc75ba1e1949d3a87 Jul 15 20:34:14.657605 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.657569 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:14.657724 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.657653 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:14.657787 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.657727 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:14.943729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.943678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerStarted","Data":"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca"} Jul 15 20:34:14.944239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.943757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerStarted","Data":"53aef422a8dd0f6ee4f5cd1f0113f7de03e091a84cec390cc75ba1e1949d3a87"} Jul 15 20:34:14.985960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:14.985825 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podStartSLOduration=0.985794168 podStartE2EDuration="985.794168ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:34:14.968800367 +0000 UTC m=+2131.678366620" watchObservedRunningTime="2026-07-15 20:34:14.985794168 +0000 UTC m=+2131.695360422" Jul 15 20:34:15.498101 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:15.498056 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:15.504297 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:15.504268 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:15.949443 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:15.949406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:15.950398 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:15.950375 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:16.931406 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:16.931368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:17.025981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:17.025938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:17.207256 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:17.207169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:17.301314 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:17.301264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:17.763338 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:17.763291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:17.856128 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:17.856096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:21.442426 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:21.442389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:28.026051 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.025925 2576 generic.go:356] "Generic (PLEG): container finished" podID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerID="102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca" exitCode=0 Jul 15 20:34:28.026051 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.025999 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerDied","Data":"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca"} Jul 15 20:34:28.026550 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.026443 2576 scope.go:117] "RemoveContainer" containerID="102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca" Jul 15 20:34:28.027480 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.027448 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:28.027578 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.027519 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:28.027578 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:28.027551 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:34:29.033103 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:29.033070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerStarted","Data":"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d"} Jul 15 20:34:29.499107 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:29.499067 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:29.499329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:29.499252 2576 patch_prober.go:28] interesting pod/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:34:29.499329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:29.499306 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 20:34:30.499095 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:30.498584 2576 patch_prober.go:28] interesting pod/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:34:30.499095 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:30.498657 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 20:34:30.949526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:30.949494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:31.008662 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:31.008621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:34:31.499258 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:31.498731 2576 patch_prober.go:28] interesting pod/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:34:31.499258 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:31.498798 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 20:34:32.498430 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:32.498387 2576 patch_prober.go:28] interesting pod/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 20:34:32.498714 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:32.498680 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 20:34:33.502497 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:33.502463 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:34:33.503609 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:34:33.503586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:35:05.156696 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:05.156657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:09.766096 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:09.766058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:14.830774 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:14.830741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:51.795663 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:51.795630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:52.054634 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:52.054554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:57.899682 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:57.899641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:35:57.970334 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:35:57.970297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:02.051997 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:02.051957 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:36:02.082905 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:02.082869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:02.322533 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:02.322441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:02.482889 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:02.482836 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" containerID="cri-o://3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d" gracePeriod=30 Jul 15 20:36:03.423496 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:03.423465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:03.504981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:03.504944 2576 patch_prober.go:28] interesting pod/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:36:03.505159 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:03.505000 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:36:03.518427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:03.518396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:03.661077 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:03.661041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:04.408256 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:04.408219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:04.539912 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:04.539877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:06.033668 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:06.033635 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:07.744416 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.744336 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:36:07.860275 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860275 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46vsl\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860413 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860489 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860781 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860781 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860566 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy\") pod \"894c44bb-18a3-4e11-9a77-9842be517c1e\" (UID: \"894c44bb-18a3-4e11-9a77-9842be517c1e\") " Jul 15 20:36:07.860781 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860681 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:36:07.860781 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.860703 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:36:07.861103 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.861077 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894c44bb-18a3-4e11-9a77-9842be517c1e-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.861303 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.861288 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.861469 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.861055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:36:07.861469 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.861099 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:36:07.861645 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.861609 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:36:07.862756 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.862732 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:36:07.862900 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.862878 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:36:07.863058 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.863027 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl" (OuterVolumeSpecName: "kube-api-access-46vsl") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "kube-api-access-46vsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:36:07.863239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.863219 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "894c44bb-18a3-4e11-9a77-9842be517c1e" (UID: "894c44bb-18a3-4e11-9a77-9842be517c1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:36:07.962556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962513 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962547 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vsl\" (UniqueName: \"kubernetes.io/projected/894c44bb-18a3-4e11-9a77-9842be517c1e-kube-api-access-46vsl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962557 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962565 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962574 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962957 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962583 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:07.962957 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:07.962591 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894c44bb-18a3-4e11-9a77-9842be517c1e-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:36:08.516566 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.516524 2576 generic.go:356] "Generic (PLEG): container finished" podID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerID="3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d" exitCode=0 Jul 15 20:36:08.516566 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.516566 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" Jul 15 20:36:08.516808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.516600 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerDied","Data":"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d"} Jul 15 20:36:08.516808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.516665 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh" event={"ID":"894c44bb-18a3-4e11-9a77-9842be517c1e","Type":"ContainerDied","Data":"53aef422a8dd0f6ee4f5cd1f0113f7de03e091a84cec390cc75ba1e1949d3a87"} Jul 15 20:36:08.516808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.516695 2576 scope.go:117] "RemoveContainer" containerID="3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d" Jul 15 20:36:08.527187 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.527164 2576 scope.go:117] "RemoveContainer" containerID="102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca" Jul 15 20:36:08.535786 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.535762 2576 scope.go:117] "RemoveContainer" containerID="3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d" Jul 15 20:36:08.536223 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:36:08.536200 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d\": container with ID starting with 3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d not found: ID does not exist" containerID="3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d" Jul 15 20:36:08.536308 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.536233 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d"} err="failed to get container status \"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d\": rpc error: code = NotFound desc = could not find container \"3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d\": container with ID starting with 3ba48aa00e1258e1390a908f615bd5e48009e62f7a1e7cd23eb8883cb93d995d not found: ID does not exist" Jul 15 20:36:08.536308 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.536253 2576 scope.go:117] "RemoveContainer" containerID="102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca" Jul 15 20:36:08.536544 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:36:08.536526 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca\": container with ID starting with 102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca not found: ID does not exist" containerID="102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca" Jul 15 20:36:08.536593 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.536551 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca"} err="failed to get container status \"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca\": rpc error: code = NotFound desc = could not find container \"102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca\": container with ID starting with 102e5cad8597e27379d804a7a2ec14ea3c1fa28d20aaaf8e5dbbf1fa5100eeca not found: ID does not exist" Jul 15 20:36:08.541546 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.541513 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:36:08.545810 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:08.545776 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2bf8d569-openshift-default-84f77866c9-bqbwh"] Jul 15 20:36:09.848388 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:09.848356 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" path="/var/lib/kubelet/pods/894c44bb-18a3-4e11-9a77-9842be517c1e/volumes" Jul 15 20:36:12.020473 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:12.020438 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:12.106358 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:12.106318 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:12.707423 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:12.707393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:18.704451 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:18.704373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:18.823297 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:18.823263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:18.915547 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:18.915504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:29.450083 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:29.450036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:29.859515 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:29.859459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:50.819097 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:50.819061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:56.503942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:56.503889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:56.573074 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:56.573037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:56.853838 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:56.853782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:57.158712 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.158624 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:36:57.158988 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.158976 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.159049 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.158989 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.159049 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.159021 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.159049 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.159030 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.159151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.159088 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.159151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.159097 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="894c44bb-18a3-4e11-9a77-9842be517c1e" containerName="istio-proxy" Jul 15 20:36:57.161966 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.161944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.164740 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.164714 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b06d27-openshift-default-dockercfg-mmjcd\"" Jul 15 20:36:57.173709 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.173677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:36:57.184292 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.184239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:57.298990 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.298949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.298990 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.298990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299463 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.299463 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.299335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcxl\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.357691 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.357654 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:57.400437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400659 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400659 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400659 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400659 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400659 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcxl\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400895 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.400895 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.400739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.401148 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.401115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.401289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.401164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.401289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.401241 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.401448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.401288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.401448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.401288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.403002 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.402979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.403122 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.403094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.409295 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.409218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.409459 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.409436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcxl\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl\") pod \"e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.473258 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.473215 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:57.621425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.621394 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:36:57.623199 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:36:57.623164 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44681425_9eb0_4ed0_bda9_815d7db03159.slice/crio-49657df6c30362785267b2702eb00261c92e0f872876dcc88987d37778c7f2bb WatchSource:0}: Error finding container 49657df6c30362785267b2702eb00261c92e0f872876dcc88987d37778c7f2bb: Status 404 returned error can't find the container with id 49657df6c30362785267b2702eb00261c92e0f872876dcc88987d37778c7f2bb Jul 15 20:36:57.625395 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.625350 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:36:57.625502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.625452 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:36:57.625564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.625500 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:36:57.751182 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.751135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerStarted","Data":"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169"} Jul 15 20:36:57.751393 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.751212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerStarted","Data":"49657df6c30362785267b2702eb00261c92e0f872876dcc88987d37778c7f2bb"} Jul 15 20:36:57.772686 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:57.772622 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podStartSLOduration=0.77260191 podStartE2EDuration="772.60191ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:36:57.771025224 +0000 UTC m=+2294.480591574" watchObservedRunningTime="2026-07-15 20:36:57.77260191 +0000 UTC m=+2294.482168161" Jul 15 20:36:58.473470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:58.473433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:58.478554 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:58.478525 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:58.755911 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:58.755822 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:58.756856 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:58.756835 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:36:59.732649 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:59.732616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:59.833152 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:59.833118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:36:59.952788 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:36:59.952751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:00.124737 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:00.124671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:00.508372 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:00.508285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:00.608444 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:00.608408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:04.251883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.251854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:04.614285 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.614250 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:04.881227 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.881139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:37:04.884908 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.884887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.887914 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.887889 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e45884-openshift-default-dockercfg-tbkvj\"" Jul 15 20:37:04.901469 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.901428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:37:04.916570 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.916540 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:04.969438 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.969388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.969868 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.969838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.969883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tgn\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.969921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.970067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.970136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.970168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.970219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:04.970371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:04.970259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tgn\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071461 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071435 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071844 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071755 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071898 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.071952 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.071909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.072169 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.072148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.072271 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.072244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.073947 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.073927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.074286 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.074264 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.080942 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.080913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.081154 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.081127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tgn\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn\") pod \"e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.196870 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.196770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:05.199643 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.199615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:05.351315 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.351285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:37:05.353480 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:37:05.353451 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6089c804_01ca_46f8_a8f0_89901c3a2839.slice/crio-f131c178e4176c9aa2b8896f6d7780f91c8792f98317311b85a2a11b85b6b00b WatchSource:0}: Error finding container f131c178e4176c9aa2b8896f6d7780f91c8792f98317311b85a2a11b85b6b00b: Status 404 returned error can't find the container with id f131c178e4176c9aa2b8896f6d7780f91c8792f98317311b85a2a11b85b6b00b Jul 15 20:37:05.356064 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.356027 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:05.356175 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.356116 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:05.356239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.356194 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:05.799193 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.799159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerStarted","Data":"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331"} Jul 15 20:37:05.799397 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.799221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerStarted","Data":"f131c178e4176c9aa2b8896f6d7780f91c8792f98317311b85a2a11b85b6b00b"} Jul 15 20:37:05.819803 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:05.819721 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podStartSLOduration=1.819695956 podStartE2EDuration="1.819695956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:37:05.818347483 +0000 UTC m=+2302.527913745" watchObservedRunningTime="2026-07-15 20:37:05.819695956 +0000 UTC m=+2302.529262209" Jul 15 20:37:06.197872 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:06.197821 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:06.210669 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:06.210638 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:06.806079 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:06.806049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:06.812885 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:06.812845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:07.773327 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:07.773292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:07.870727 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:07.870690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:07.991462 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:07.991418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:08.192363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:08.192317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:08.435680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:08.435651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:08.536995 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:08.536899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:12.279491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:12.279459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:13.757784 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.757747 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:37:13.758544 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.757804 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 20:37:13.845669 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.845631 2576 generic.go:356] "Generic (PLEG): container finished" podID="44681425-9eb0-4ed0-bda9-815d7db03159" containerID="7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169" exitCode=0 Jul 15 20:37:13.849233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.849205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerDied","Data":"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169"} Jul 15 20:37:13.849633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.849617 2576 scope.go:117] "RemoveContainer" containerID="7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169" Jul 15 20:37:13.850691 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.850655 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:13.850768 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.850732 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:13.850807 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:13.850768 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:14.852887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:14.852851 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerStarted","Data":"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0"} Jul 15 20:37:15.474441 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:15.474356 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:37:15.474628 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:15.474585 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:37:15.474676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:15.474637 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 20:37:16.473754 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:16.473720 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:37:16.474182 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:16.473787 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 20:37:17.474695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:17.474658 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:37:17.475101 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:17.474729 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 20:37:18.474435 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:18.474398 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 20:37:18.474620 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:18.474476 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 20:37:19.478125 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:19.478093 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:37:19.478564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:19.478365 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:37:19.479255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:19.479239 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:37:20.884568 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.884530 2576 generic.go:356] "Generic (PLEG): container finished" podID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerID="fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331" exitCode=0 Jul 15 20:37:20.885118 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.884614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerDied","Data":"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331"} Jul 15 20:37:20.885277 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.885256 2576 scope.go:117] "RemoveContainer" containerID="fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331" Jul 15 20:37:20.886357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.886324 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:20.886442 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.886424 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:20.886500 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:20.886473 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:37:21.142033 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:21.141912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:21.526886 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:21.526796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:21.598949 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:21.598910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:21.806085 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:21.805930 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:21.894593 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:21.894548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerStarted","Data":"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c"} Jul 15 20:37:22.197317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:22.197277 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:22.198995 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:22.198954 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 20:37:22.199317 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:22.199265 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 20:37:23.197758 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:23.197711 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 20:37:23.198291 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:23.197784 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 20:37:24.198262 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:24.198219 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 20:37:24.199551 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:24.199512 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 20:37:25.177424 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:25.177375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:25.197665 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:25.197622 2576 patch_prober.go:28] interesting pod/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 20:37:25.197870 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:25.197689 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 20:37:25.278650 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:25.278615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:26.201545 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:26.201513 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:26.201928 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:26.201906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:26.203151 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:26.203130 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:37:29.480694 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:29.480655 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:29.578834 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:29.578800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:37:38.363934 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:37:38.363841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:06.060261 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:06.060225 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:07.020642 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:07.020609 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:07.741989 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:07.741946 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:07.843168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:07.843131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:17.565604 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:17.565564 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:22.196251 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:22.196206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:27.945189 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:27.945159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:28.045281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:28.045243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:29.299960 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:29.299909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:32.859299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:32.859263 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:38:32.859796 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:32.859524 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" containerID="cri-o://0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0" gracePeriod=30 Jul 15 20:38:32.889099 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:32.889048 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:33.146947 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:33.146860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:33.517128 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:33.517033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:33.917042 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:33.916977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:33.991946 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:33.991911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:34.480716 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:34.480678 2576 patch_prober.go:28] interesting pod/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:38:34.480959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:34.480735 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:38:38.126603 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.126529 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:38:38.233483 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkcxl\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233539 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233564 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233670 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233661 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233693 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233749 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data\") pod \"44681425-9eb0-4ed0-bda9-815d7db03159\" (UID: \"44681425-9eb0-4ed0-bda9-815d7db03159\") " Jul 15 20:38:38.233968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233922 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:38:38.233968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.233937 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:38:38.234221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.234059 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.234221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.234077 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.234221 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.234177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:38:38.234380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.234339 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:38:38.234814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.234789 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data" (OuterVolumeSpecName: "istio-data") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:38:38.236115 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.236074 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token" (OuterVolumeSpecName: "istio-token") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:38:38.236215 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.236140 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:38:38.236319 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.236299 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:38:38.236369 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.236309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl" (OuterVolumeSpecName: "kube-api-access-jkcxl") pod "44681425-9eb0-4ed0-bda9-815d7db03159" (UID: "44681425-9eb0-4ed0-bda9-815d7db03159"). InnerVolumeSpecName "kube-api-access-jkcxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:38:38.273953 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.273919 2576 generic.go:356] "Generic (PLEG): container finished" podID="44681425-9eb0-4ed0-bda9-815d7db03159" containerID="0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0" exitCode=0 Jul 15 20:38:38.274163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.273964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" Jul 15 20:38:38.274163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.274033 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerDied","Data":"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0"} Jul 15 20:38:38.274163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.274099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz" event={"ID":"44681425-9eb0-4ed0-bda9-815d7db03159","Type":"ContainerDied","Data":"49657df6c30362785267b2702eb00261c92e0f872876dcc88987d37778c7f2bb"} Jul 15 20:38:38.274163 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.274133 2576 scope.go:117] "RemoveContainer" containerID="0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0" Jul 15 20:38:38.283489 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.283471 2576 scope.go:117] "RemoveContainer" containerID="7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169" Jul 15 20:38:38.292537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.292513 2576 scope.go:117] "RemoveContainer" containerID="0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0" Jul 15 20:38:38.292856 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:38:38.292839 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0\": container with ID starting with 0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0 not found: ID does not exist" containerID="0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0" Jul 15 20:38:38.292914 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.292864 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0"} err="failed to get container status \"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0\": rpc error: code = NotFound desc = could not find container \"0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0\": container with ID starting with 0322259aebf0868f53c55532ce845abc9186ec77ecfc46727440d034a30e34c0 not found: ID does not exist" Jul 15 20:38:38.292914 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.292882 2576 scope.go:117] "RemoveContainer" containerID="7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169" Jul 15 20:38:38.293133 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:38:38.293114 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169\": container with ID starting with 7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169 not found: ID does not exist" containerID="7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169" Jul 15 20:38:38.293179 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.293139 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169"} err="failed to get container status \"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169\": rpc error: code = NotFound desc = could not find container \"7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169\": container with ID starting with 7d9ffd744e7c7f952b82f2e0757219465635d454b02838b6ac9e8ccafe41c169 not found: ID does not exist" Jul 15 20:38:38.297435 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.297398 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:38:38.304588 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.304564 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b06d27-openshift-default-c48499755-mvfvz"] Jul 15 20:38:38.335557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335518 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44681425-9eb0-4ed0-bda9-815d7db03159-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335550 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44681425-9eb0-4ed0-bda9-815d7db03159-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335557 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335562 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335572 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335586 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335596 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44681425-9eb0-4ed0-bda9-815d7db03159-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:38.335782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:38.335607 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcxl\" (UniqueName: \"kubernetes.io/projected/44681425-9eb0-4ed0-bda9-815d7db03159-kube-api-access-jkcxl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:38:39.848775 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:39.848739 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" path="/var/lib/kubelet/pods/44681425-9eb0-4ed0-bda9-815d7db03159/volumes" Jul 15 20:38:43.826981 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:43.826942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:43.864674 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:43.864647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:38:43.871171 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:43.871141 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:38:48.336729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:48.336686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:54.396114 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:54.396058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:54.965371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:54.965321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:55.062258 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:55.062218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:55.181815 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:55.181777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:55.283448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:55.283359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:59.045749 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.045712 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:38:59.046293 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.045938 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" containerID="cri-o://ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c" gracePeriod=30 Jul 15 20:38:59.069527 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.069493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:59.302792 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.302703 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:59.667162 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.667126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:38:59.957685 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.957598 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:38:59.957973 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.957959 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:38:59.958055 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.957974 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:38:59.958118 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.958064 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:38:59.958118 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.958085 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:38:59.961131 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.961111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:38:59.963728 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.963708 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4fa9c7-openshift-default-dockercfg-lrkxn\"" Jul 15 20:38:59.972932 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.972907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:38:59.985812 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:38:59.985776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:00.035363 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmm2m\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035541 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.035776 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.035672 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137053 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.136987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137539 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmm2m\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.137974 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.137955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.138072 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.138044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.138273 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.138079 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.138434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.138411 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.138434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.138421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.140438 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.140406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.140536 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.140436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.142075 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.142049 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:00.145943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.145915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmm2m\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.146128 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.146105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token\") pod \"e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.271757 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.271654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:00.424070 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.423970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:39:00.426714 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:39:00.426682 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a39c5e_e5d6_4e8a_bb88_5c02f99ecb21.slice/crio-83bb43738504f8fc8a33ba56f732d8504733f2c2ba76686c652ff7b275e39e68 WatchSource:0}: Error finding container 83bb43738504f8fc8a33ba56f732d8504733f2c2ba76686c652ff7b275e39e68: Status 404 returned error can't find the container with id 83bb43738504f8fc8a33ba56f732d8504733f2c2ba76686c652ff7b275e39e68 Jul 15 20:39:00.429228 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.429189 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:00.429304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.429290 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:00.429356 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:00.429338 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:01.394917 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:01.394881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerStarted","Data":"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4"} Jul 15 20:39:01.395372 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:01.394934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerStarted","Data":"83bb43738504f8fc8a33ba56f732d8504733f2c2ba76686c652ff7b275e39e68"} Jul 15 20:39:01.418512 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:01.418453 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podStartSLOduration=2.4184353659999998 podStartE2EDuration="2.418435366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:39:01.417555692 +0000 UTC m=+2418.127121944" watchObservedRunningTime="2026-07-15 20:39:01.418435366 +0000 UTC m=+2418.128001617" Jul 15 20:39:02.272352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:02.272305 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:02.277135 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:02.277107 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:02.400304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:02.400262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:02.401285 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:02.401264 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:03.363822 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:03.363785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:03.462488 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:03.462450 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:03.656778 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:03.656672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:04.111788 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.111756 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:04.215112 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.215076 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:04.314818 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.314739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:04.315743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.315724 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:39:04.375137 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375103 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375153 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375212 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8tgn\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375239 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375304 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375298 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375362 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375417 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:04.375607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token\") pod \"6089c804-01ca-46f8-a8f0-89901c3a2839\" (UID: \"6089c804-01ca-46f8-a8f0-89901c3a2839\") " Jul 15 20:39:04.375607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375519 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:04.375830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375692 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:04.375830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375773 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:39:04.375830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375790 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.375830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375811 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.375830 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.375825 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.376336 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.376308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data" (OuterVolumeSpecName: "istio-data") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:04.377524 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.377497 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn" (OuterVolumeSpecName: "kube-api-access-b8tgn") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "kube-api-access-b8tgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:39:04.377633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.377557 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:39:04.377633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.377562 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:04.377633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.377625 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token" (OuterVolumeSpecName: "istio-token") pod "6089c804-01ca-46f8-a8f0-89901c3a2839" (UID: "6089c804-01ca-46f8-a8f0-89901c3a2839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:39:04.412235 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.412199 2576 generic.go:356] "Generic (PLEG): container finished" podID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerID="ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c" exitCode=0 Jul 15 20:39:04.412429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.412252 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" Jul 15 20:39:04.412429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.412282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerDied","Data":"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c"} Jul 15 20:39:04.412429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.412337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2" event={"ID":"6089c804-01ca-46f8-a8f0-89901c3a2839","Type":"ContainerDied","Data":"f131c178e4176c9aa2b8896f6d7780f91c8792f98317311b85a2a11b85b6b00b"} Jul 15 20:39:04.412429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.412359 2576 scope.go:117] "RemoveContainer" containerID="ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c" Jul 15 20:39:04.425506 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.425478 2576 scope.go:117] "RemoveContainer" containerID="fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331" Jul 15 20:39:04.433895 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.433874 2576 scope.go:117] "RemoveContainer" containerID="ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c" Jul 15 20:39:04.434230 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:39:04.434209 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c\": container with ID starting with ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c not found: ID does not exist" containerID="ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c" Jul 15 20:39:04.434286 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.434238 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c"} err="failed to get container status \"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c\": rpc error: code = NotFound desc = could not find container \"ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c\": container with ID starting with ef9fad104bc2433b69b5a9ee427ba61e60bb5f9b968f5b6ed9dc1f50f4a0ad9c not found: ID does not exist" Jul 15 20:39:04.434286 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.434255 2576 scope.go:117] "RemoveContainer" containerID="fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331" Jul 15 20:39:04.434513 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:39:04.434494 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331\": container with ID starting with fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331 not found: ID does not exist" containerID="fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331" Jul 15 20:39:04.434559 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.434521 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331"} err="failed to get container status \"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331\": rpc error: code = NotFound desc = could not find container \"fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331\": container with ID starting with fc3c8a8fc276ddf1f83833ec6357f668a5f2a56b81af9b421e2652e2c2afc331 not found: ID does not exist" Jul 15 20:39:04.437965 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.437935 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:39:04.442226 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.442203 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e45884-openshift-default-857f5c8f55-842h2"] Jul 15 20:39:04.476824 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476784 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.476824 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476824 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.477289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476838 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tgn\" (UniqueName: \"kubernetes.io/projected/6089c804-01ca-46f8-a8f0-89901c3a2839-kube-api-access-b8tgn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.477289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476853 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6089c804-01ca-46f8-a8f0-89901c3a2839-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.477289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476867 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.477289 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.476880 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6089c804-01ca-46f8-a8f0-89901c3a2839-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:04.503324 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.503289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:04.564456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:04.564429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:05.848562 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:05.848481 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" path="/var/lib/kubelet/pods/6089c804-01ca-46f8-a8f0-89901c3a2839/volumes" Jul 15 20:39:07.889636 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:07.889584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:08.361722 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.361636 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:08.667546 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667456 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:39:08.667812 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667799 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:39:08.667883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667812 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:39:08.667883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667834 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:39:08.667883 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667840 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="44681425-9eb0-4ed0-bda9-815d7db03159" containerName="istio-proxy" Jul 15 20:39:08.668059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.667902 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:39:08.672480 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.672454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.675252 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.675228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-44c9f6-openshift-default-dockercfg-xrsv7\"" Jul 15 20:39:08.690313 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.690271 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:39:08.704058 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.703998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:08.718683 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.718643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.718935 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.718916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719099 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719327 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719435 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbgg\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719551 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719671 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.719791 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.719779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.820943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.820902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.820943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.820945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.820973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbgg\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821427 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821577 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821477 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821643 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.821839 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.822044 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.821996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.822044 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.822000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.823532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.823511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.823884 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.823862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.830620 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.830590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.830869 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.830847 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbgg\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg\") pod \"e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:08.983691 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:08.983581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:09.177085 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.177046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:39:09.180105 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:39:09.180074 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dbdc17_9447_4717_832a_8c3a0afe0504.slice/crio-a9e18bebe3fede940f3230d15a0fb7c5cbefb391913cc719697ddf673098bf25 WatchSource:0}: Error finding container a9e18bebe3fede940f3230d15a0fb7c5cbefb391913cc719697ddf673098bf25: Status 404 returned error can't find the container with id a9e18bebe3fede940f3230d15a0fb7c5cbefb391913cc719697ddf673098bf25 Jul 15 20:39:09.182380 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.182341 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:09.182470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.182423 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:09.182470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.182455 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:09.242920 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.242884 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:09.440288 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.440244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerStarted","Data":"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5"} Jul 15 20:39:09.440493 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.440309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerStarted","Data":"a9e18bebe3fede940f3230d15a0fb7c5cbefb391913cc719697ddf673098bf25"} Jul 15 20:39:09.464270 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.464222 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podStartSLOduration=1.4642031549999999 podStartE2EDuration="1.464203155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:39:09.462575566 +0000 UTC m=+2426.172141819" watchObservedRunningTime="2026-07-15 20:39:09.464203155 +0000 UTC m=+2426.173769408" Jul 15 20:39:09.984532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:09.984487 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:11.000740 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.000706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:11.400770 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.400675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:11.450032 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.449975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:11.451077 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.451057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:11.488036 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.483334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:11.556269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.556231 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:11.988109 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:11.988065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:12.093039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:12.091847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:12.188723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:12.188681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:15.895120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:15.894993 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:16.478113 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.478074 2576 generic.go:356] "Generic (PLEG): container finished" podID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerID="04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4" exitCode=0 Jul 15 20:39:16.478305 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.478150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerDied","Data":"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4"} Jul 15 20:39:16.478574 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.478557 2576 scope.go:117] "RemoveContainer" containerID="04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4" Jul 15 20:39:16.479547 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.479514 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:16.479640 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.479583 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:16.479640 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:16.479615 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:17.400866 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:17.400818 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:17.487292 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:17.486276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerStarted","Data":"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505"} Jul 15 20:39:18.272522 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:18.272478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:18.273706 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:18.273678 2576 patch_prober.go:28] interesting pod/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:39:18.273835 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:18.273729 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 15 20:39:19.273000 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:19.272958 2576 patch_prober.go:28] interesting pod/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:39:19.273456 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:19.273063 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 15 20:39:20.272984 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:20.272944 2576 patch_prober.go:28] interesting pod/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:39:20.273193 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:20.273002 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 15 20:39:21.272958 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:21.272924 2576 patch_prober.go:28] interesting pod/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 15 20:39:21.273164 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:21.272981 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 15 20:39:22.276064 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:22.276029 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:22.276528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:22.276345 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:22.277110 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:22.277094 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:24.618414 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:24.618377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:24.719669 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:24.719637 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:27.491269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:27.491223 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 20:39:27.491676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:27.491304 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 20:39:29.559502 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.559463 2576 generic.go:356] "Generic (PLEG): container finished" podID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerID="b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5" exitCode=0 Jul 15 20:39:29.560057 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.559541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerDied","Data":"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5"} Jul 15 20:39:29.560131 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.560107 2576 scope.go:117] "RemoveContainer" containerID="b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5" Jul 15 20:39:29.561181 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.561151 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:29.561248 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.561226 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:29.561284 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:29.561259 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:39:30.566416 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:30.566381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerStarted","Data":"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea"} Jul 15 20:39:30.984281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:30.984235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:30.984503 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:30.984436 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:39:30.984580 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:30.984495 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 20:39:31.984698 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:31.984649 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:39:31.985245 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:31.984727 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 20:39:32.935867 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:32.935831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:32.984724 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:32.984680 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:39:32.985260 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:32.984756 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 20:39:33.035621 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:33.035589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:33.984300 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:33.984259 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 20:39:33.984535 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:33.984329 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 20:39:34.636951 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.636917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:34.737940 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.737906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:34.906949 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.906855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:34.989827 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.989792 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:34.990232 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.990210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:34.991105 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:34.991085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:39:39.094706 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:39.094673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:41.923404 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:41.923372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:42.024707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:42.024675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:44.751084 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:44.751042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:44.815695 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:44.815662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:49.789915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:49.789873 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:39:49.790423 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:49.790312 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" containerID="cri-o://ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505" gracePeriod=30 Jul 15 20:39:49.819964 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:49.819927 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:50.038973 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:50.038938 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:50.804723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:50.804690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:50.905378 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:50.905343 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:39:52.279038 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:52.278978 2576 patch_prober.go:28] interesting pod/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:39:52.279445 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:52.279090 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:39:55.046656 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.046611 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:55.055196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055109 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055196 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055199 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055258 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055292 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055408 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmm2m\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055423 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055457 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo\") pod \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\" (UID: \"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21\") " Jul 15 20:39:55.055658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055473 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:55.055658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055528 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:55.055658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055602 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:55.055906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055702 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:39:55.055906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055803 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.055906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055820 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.055906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055834 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.055906 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.055847 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.056189 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.056166 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data" (OuterVolumeSpecName: "istio-data") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:55.057623 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.057587 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token" (OuterVolumeSpecName: "istio-token") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:39:55.057743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.057700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:39:55.058022 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.057987 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:39:55.058108 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.058000 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m" (OuterVolumeSpecName: "kube-api-access-fmm2m") pod "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" (UID: "24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21"). InnerVolumeSpecName "kube-api-access-fmm2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:39:55.156504 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.156465 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.156504 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.156498 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmm2m\" (UniqueName: \"kubernetes.io/projected/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-kube-api-access-fmm2m\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.156504 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.156509 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.156747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.156520 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.156747 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.156528 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:39:55.699547 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.699510 2576 generic.go:356] "Generic (PLEG): container finished" podID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerID="ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505" exitCode=0 Jul 15 20:39:55.699729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.699562 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" Jul 15 20:39:55.699729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.699594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerDied","Data":"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505"} Jul 15 20:39:55.699729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.699659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr" event={"ID":"24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21","Type":"ContainerDied","Data":"83bb43738504f8fc8a33ba56f732d8504733f2c2ba76686c652ff7b275e39e68"} Jul 15 20:39:55.699729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.699692 2576 scope.go:117] "RemoveContainer" containerID="ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505" Jul 15 20:39:55.709370 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.709348 2576 scope.go:117] "RemoveContainer" containerID="04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4" Jul 15 20:39:55.717729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.717708 2576 scope.go:117] "RemoveContainer" containerID="ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505" Jul 15 20:39:55.718061 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:39:55.718036 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505\": container with ID starting with ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505 not found: ID does not exist" containerID="ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505" Jul 15 20:39:55.718167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.718074 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505"} err="failed to get container status \"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505\": rpc error: code = NotFound desc = could not find container \"ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505\": container with ID starting with ba4ccf5c732a9ff6042393debec512aa3b844644d998c0873fbba524118f1505 not found: ID does not exist" Jul 15 20:39:55.718167 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.718100 2576 scope.go:117] "RemoveContainer" containerID="04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4" Jul 15 20:39:55.718378 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:39:55.718362 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4\": container with ID starting with 04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4 not found: ID does not exist" containerID="04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4" Jul 15 20:39:55.718437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.718389 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4"} err="failed to get container status \"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4\": rpc error: code = NotFound desc = could not find container \"04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4\": container with ID starting with 04459305dfd780ebd6a5e84d2cd4248578a0e9372f72dbfef6a2f79fe98c23d4 not found: ID does not exist" Jul 15 20:39:55.739860 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.739825 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:39:55.746943 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.746913 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4fa9c7-openshift-default-844dd65859-9nxbr"] Jul 15 20:39:55.849106 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:39:55.849065 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" path="/var/lib/kubelet/pods/24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21/volumes" Jul 15 20:40:00.688428 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:00.688394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:05.318492 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:05.318454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:11.138442 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:11.138406 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:11.202300 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:11.202269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:15.925748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:15.925706 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:40:15.926383 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:15.925959 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" containerID="cri-o://297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea" gracePeriod=30 Jul 15 20:40:16.094568 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.094518 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:16.195815 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.195721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:16.549578 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.549498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:16.842680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.842640 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:40:16.843115 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843094 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843115 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843116 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843277 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843154 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843277 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843163 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843277 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843176 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:40:16.843277 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843184 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:40:16.843464 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843302 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843464 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843317 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a39c5e-e5d6-4e8a-bb88-5c02f99ecb21" containerName="istio-proxy" Jul 15 20:40:16.843464 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.843327 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6089c804-01ca-46f8-a8f0-89901c3a2839" containerName="istio-proxy" Jul 15 20:40:16.846181 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.846152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.848917 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.848894 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-724347-openshift-default-dockercfg-9ftk2\"" Jul 15 20:40:16.858707 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.858680 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:40:16.873492 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.873459 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:16.950208 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950226 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950273 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvrq\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:16.950657 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:16.950607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.030841 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.030795 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:17.051983 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.051945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.051983 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.051984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvrq\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052240 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052426 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052705 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.052939 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.052921 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.054505 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.054483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.054637 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.054616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.061686 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.061661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.061844 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.061819 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvrq\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq\") pod \"e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.157771 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.157666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:17.300741 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.300709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:40:17.302411 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:40:17.302383 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebf9f4a_2945_428d_bb76_bed09448eff5.slice/crio-7d1b6c37105088cf30b132bf4e79a4f88a3326188c615fc1012877ce4932e2b1 WatchSource:0}: Error finding container 7d1b6c37105088cf30b132bf4e79a4f88a3326188c615fc1012877ce4932e2b1: Status 404 returned error can't find the container with id 7d1b6c37105088cf30b132bf4e79a4f88a3326188c615fc1012877ce4932e2b1 Jul 15 20:40:17.304527 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.304492 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:17.304622 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.304566 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:17.304622 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.304601 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:17.814097 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.814059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerStarted","Data":"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb"} Jul 15 20:40:17.814309 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.814125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerStarted","Data":"7d1b6c37105088cf30b132bf4e79a4f88a3326188c615fc1012877ce4932e2b1"} Jul 15 20:40:17.836744 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:17.836693 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podStartSLOduration=1.836673108 podStartE2EDuration="1.836673108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:40:17.834851303 +0000 UTC m=+2494.544417556" watchObservedRunningTime="2026-07-15 20:40:17.836673108 +0000 UTC m=+2494.546239361" Jul 15 20:40:18.158891 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:18.158845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:18.163843 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:18.163810 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:18.819120 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:18.819086 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:18.820046 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:18.820023 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:19.743676 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:19.743641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:19.899214 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:19.899168 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:19.992887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:19.992846 2576 patch_prober.go:28] interesting pod/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:40:19.993085 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:19.992946 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:40:19.999558 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:19.999472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:20.406365 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:20.406330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:20.509588 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:20.509550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:20.610216 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:20.610180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:21.177183 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.177152 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:40:21.292767 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292662 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.292767 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.292767 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292734 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.292767 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292753 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292782 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wbgg\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292878 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292922 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.292970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket\") pod \"55dbdc17-9447-4717-832a-8c3a0afe0504\" (UID: \"55dbdc17-9447-4717-832a-8c3a0afe0504\") " Jul 15 20:40:21.293494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.293217 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:40:21.293494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.293338 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.293494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.293456 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:40:21.293656 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.293565 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:40:21.293812 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.293788 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:40:21.294258 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.294228 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data" (OuterVolumeSpecName: "istio-data") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:40:21.295291 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.295256 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg" (OuterVolumeSpecName: "kube-api-access-6wbgg") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "kube-api-access-6wbgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:40:21.295400 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.295307 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:40:21.295400 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.295334 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:40:21.295643 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.295622 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token" (OuterVolumeSpecName: "istio-token") pod "55dbdc17-9447-4717-832a-8c3a0afe0504" (UID: "55dbdc17-9447-4717-832a-8c3a0afe0504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:40:21.394496 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394456 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55dbdc17-9447-4717-832a-8c3a0afe0504-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394496 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394497 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wbgg\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-kube-api-access-6wbgg\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394513 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394526 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394538 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394551 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394564 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55dbdc17-9447-4717-832a-8c3a0afe0504-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.394715 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.394577 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55dbdc17-9447-4717-832a-8c3a0afe0504-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:40:21.839834 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.839800 2576 generic.go:356] "Generic (PLEG): container finished" podID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerID="297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea" exitCode=0 Jul 15 20:40:21.840039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.839850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" Jul 15 20:40:21.840039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.839877 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerDied","Data":"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea"} Jul 15 20:40:21.840039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.839934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q" event={"ID":"55dbdc17-9447-4717-832a-8c3a0afe0504","Type":"ContainerDied","Data":"a9e18bebe3fede940f3230d15a0fb7c5cbefb391913cc719697ddf673098bf25"} Jul 15 20:40:21.840039 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.839958 2576 scope.go:117] "RemoveContainer" containerID="297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea" Jul 15 20:40:21.852482 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.852452 2576 scope.go:117] "RemoveContainer" containerID="b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5" Jul 15 20:40:21.861764 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.861583 2576 scope.go:117] "RemoveContainer" containerID="297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea" Jul 15 20:40:21.862043 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:40:21.861987 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea\": container with ID starting with 297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea not found: ID does not exist" containerID="297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea" Jul 15 20:40:21.862168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.862052 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea"} err="failed to get container status \"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea\": rpc error: code = NotFound desc = could not find container \"297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea\": container with ID starting with 297d5d759ec4bc477ef1cfadde2922579926f8bb8def5cb4ea66fd0fde2c40ea not found: ID does not exist" Jul 15 20:40:21.862168 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.862078 2576 scope.go:117] "RemoveContainer" containerID="b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5" Jul 15 20:40:21.862417 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:40:21.862396 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5\": container with ID starting with b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5 not found: ID does not exist" containerID="b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5" Jul 15 20:40:21.862459 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.862422 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5"} err="failed to get container status \"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5\": rpc error: code = NotFound desc = could not find container \"b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5\": container with ID starting with b9a29a034464a528d6b764c4fecaa510cc21b66a7a210a35400011f37837abe5 not found: ID does not exist" Jul 15 20:40:21.868915 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.868885 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:40:21.870796 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:21.870773 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-44c9f6-openshift-default-7569d7d449-5l89q"] Jul 15 20:40:23.848730 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:23.848696 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" path="/var/lib/kubelet/pods/55dbdc17-9447-4717-832a-8c3a0afe0504/volumes" Jul 15 20:40:24.304236 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.304203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:24.643929 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.643889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:24.966757 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.966664 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:40:24.967278 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.967258 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:40:24.967355 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.967281 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:40:24.967417 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.967395 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:40:24.967417 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.967408 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:40:24.971894 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.971873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:24.975088 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.975068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5292f5-openshift-default-dockercfg-4wh9h\"" Jul 15 20:40:24.988640 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:24.988605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:40:25.006092 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.006043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:25.128089 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128299 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128471 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128471 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128471 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128471 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf5f\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.128597 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.128476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229498 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf5f\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229498 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229498 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.229782 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230082 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230082 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.229957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230082 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.230047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.230125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230194 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.230142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.230487 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.230464 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.232067 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.232047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.232272 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.232256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.237794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.237758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf5f\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.238059 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.238038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token\") pod \"e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.246233 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.246192 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:25.282188 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.282158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:25.438224 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.438192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:40:25.440466 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:40:25.440436 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e167b89_26ad_4ab7_892a_85816f906eeb.slice/crio-ce20311ee59470f75784c2e76c10515a25b1c6a098538ee44e3d712f74f33b53 WatchSource:0}: Error finding container ce20311ee59470f75784c2e76c10515a25b1c6a098538ee44e3d712f74f33b53: Status 404 returned error can't find the container with id ce20311ee59470f75784c2e76c10515a25b1c6a098538ee44e3d712f74f33b53 Jul 15 20:40:25.442984 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.442950 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:25.443084 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.443040 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:25.443084 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.443070 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:25.860396 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.860358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerStarted","Data":"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195"} Jul 15 20:40:25.860554 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.860421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerStarted","Data":"ce20311ee59470f75784c2e76c10515a25b1c6a098538ee44e3d712f74f33b53"} Jul 15 20:40:25.886935 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:25.886875 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podStartSLOduration=1.886852362 podStartE2EDuration="1.886852362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:40:25.886121009 +0000 UTC m=+2502.595687354" watchObservedRunningTime="2026-07-15 20:40:25.886852362 +0000 UTC m=+2502.596418615" Jul 15 20:40:26.283421 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:26.283321 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:26.289164 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:26.289131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:26.866718 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:26.866683 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:26.867694 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:26.867674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:27.848337 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:27.848302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:27.926769 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:27.926733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:28.005633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:28.005590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:28.617851 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:28.617763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:28.719513 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:28.719471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:32.341816 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:32.341775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:34.017476 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.017343 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:58882->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 15 20:40:34.017476 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.017408 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:58882->10.134.0.53:15021: read: connection reset by peer" Jul 15 20:40:34.914731 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.914637 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerID="36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb" exitCode=0 Jul 15 20:40:34.915173 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.914989 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerDied","Data":"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb"} Jul 15 20:40:34.915634 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.915617 2576 scope.go:117] "RemoveContainer" containerID="36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb" Jul 15 20:40:34.916916 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.916887 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:34.916992 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.916963 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:34.917066 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:34.916996 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:35.926714 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:35.926666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerStarted","Data":"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee"} Jul 15 20:40:36.159139 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:36.159098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:36.159323 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:36.159228 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 15 20:40:36.159323 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:36.159270 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 15 20:40:37.158433 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:37.158078 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 15 20:40:37.158433 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:37.158134 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 15 20:40:38.158353 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:38.158315 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 15 20:40:38.158552 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:38.158387 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 15 20:40:39.158767 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:39.158728 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 15 20:40:39.159315 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:39.158792 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 15 20:40:40.162520 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:40.162466 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:40.163049 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:40.163003 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:40.163828 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:40.163799 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:40:41.248244 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.248195 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:41.348518 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.348433 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:41.868804 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.868712 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:40:41.868804 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.868772 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 15 20:40:41.962146 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.962109 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerID="8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195" exitCode=0 Jul 15 20:40:41.962346 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.962188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerDied","Data":"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195"} Jul 15 20:40:41.962690 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.962674 2576 scope.go:117] "RemoveContainer" containerID="8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195" Jul 15 20:40:41.963691 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.963663 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:41.963792 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.963759 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:41.963854 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:41.963818 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 15 20:40:42.968794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:42.968760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerStarted","Data":"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4"} Jul 15 20:40:43.282802 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:43.282699 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:43.282976 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:43.282952 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:40:43.283080 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:43.283003 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 15 20:40:44.283532 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:44.283431 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:40:44.284134 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:44.284097 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 15 20:40:45.282771 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:45.282720 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:40:45.282985 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:45.282803 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 15 20:40:46.283002 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:46.282962 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 15 20:40:46.283537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:46.283083 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 15 20:40:47.287574 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:47.287539 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:47.288128 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:47.287914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:47.288808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:47.288788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:40:49.091750 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:49.091719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:49.192841 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:49.192800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:40:53.192303 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:40:53.192238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:41:12.586352 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:41:12.586309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:41:27.177171 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:41:27.177129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:43:35.671680 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:35.671593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:43:36.758723 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:36.758690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:43:43.894727 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:43.894697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:43:43.901725 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:43.901702 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:43:46.882414 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:46.882373 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:43:55.712994 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:55.712949 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:43:55.817873 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:43:55.817844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:44:01.610481 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:44:01.610439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:44:08.619419 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:44:08.619378 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:44:14.674278 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:44:14.674223 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:01.220474 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:01.220428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:02.256342 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:02.256306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:19.074614 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:19.074513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:42.005110 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:42.005067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:42.105784 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:42.105743 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:49.830112 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:49.830067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:45:49.930748 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:45:49.930707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:46:00.976181 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:46:00.976144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:46:13.519078 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:46:13.519037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:46:49.349798 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:46:49.349759 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:46:50.372361 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:46:50.372307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:10.412266 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:10.412232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:10.505814 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:10.505776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:20.332761 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:20.332722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:25.016993 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:25.016956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:30.839068 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:30.839031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:30.912262 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:30.912227 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:35.679791 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:35.679755 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:47:35.680243 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:35.679961 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" containerID="cri-o://a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee" gracePeriod=30 Jul 15 20:47:35.709577 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:35.709541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:35.926113 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:35.926071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:36.652172 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:36.652132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:36.753201 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:36.753157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:38.408024 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:38.407977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:38.463556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:38.463526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:40.165717 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:40.165681 2576 patch_prober.go:28] interesting pod/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:47:40.166129 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:40.165745 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:47:40.950376 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:40.950302 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:47:41.041047 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.040981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmvrq\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041078 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041106 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041191 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041255 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041267 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041348 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert\") pod \"3ebf9f4a-2945-428d-bb76-bed09448eff5\" (UID: \"3ebf9f4a-2945-428d-bb76-bed09448eff5\") " Jul 15 20:47:41.041528 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041486 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:47:41.041735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041527 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:47:41.041735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041548 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:47:41.041735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041702 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.041735 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041722 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.041929 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041737 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.041929 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.041836 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:47:41.042076 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.042051 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:47:41.043462 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.043430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq" (OuterVolumeSpecName: "kube-api-access-wmvrq") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "kube-api-access-wmvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:47:41.043597 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.043554 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:47:41.043597 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.043570 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:47:41.043673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.043619 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ebf9f4a-2945-428d-bb76-bed09448eff5" (UID: "3ebf9f4a-2945-428d-bb76-bed09448eff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:47:41.069185 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.069149 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerID="a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee" exitCode=0 Jul 15 20:47:41.069391 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.069206 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" Jul 15 20:47:41.069391 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.069228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerDied","Data":"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee"} Jul 15 20:47:41.069391 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.069285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8" event={"ID":"3ebf9f4a-2945-428d-bb76-bed09448eff5","Type":"ContainerDied","Data":"7d1b6c37105088cf30b132bf4e79a4f88a3326188c615fc1012877ce4932e2b1"} Jul 15 20:47:41.069391 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.069307 2576 scope.go:117] "RemoveContainer" containerID="a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee" Jul 15 20:47:41.078691 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.078670 2576 scope.go:117] "RemoveContainer" containerID="36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb" Jul 15 20:47:41.087384 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.087360 2576 scope.go:117] "RemoveContainer" containerID="a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee" Jul 15 20:47:41.087749 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:47:41.087725 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee\": container with ID starting with a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee not found: ID does not exist" containerID="a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee" Jul 15 20:47:41.087866 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.087756 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee"} err="failed to get container status \"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee\": rpc error: code = NotFound desc = could not find container \"a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee\": container with ID starting with a7dc180c963ebecb1f369fd83d55cbcc943bc11d7c1a9e219c92ec28bdb3cbee not found: ID does not exist" Jul 15 20:47:41.087866 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.087777 2576 scope.go:117] "RemoveContainer" containerID="36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb" Jul 15 20:47:41.088040 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:47:41.088001 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb\": container with ID starting with 36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb not found: ID does not exist" containerID="36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb" Jul 15 20:47:41.088108 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.088045 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb"} err="failed to get container status \"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb\": rpc error: code = NotFound desc = could not find container \"36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb\": container with ID starting with 36caf42db98a7aa3da614d83ec08159ce4a9f53452d00adb793d7a874e71bcfb not found: ID does not exist" Jul 15 20:47:41.091400 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.091373 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:47:41.095584 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.095555 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-724347-openshift-default-57b6f9d7f9-pkxq8"] Jul 15 20:47:41.142794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142756 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.142794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142787 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.142794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142796 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebf9f4a-2945-428d-bb76-bed09448eff5-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.142794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142804 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmvrq\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-kube-api-access-wmvrq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.143108 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142814 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.143108 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.142822 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebf9f4a-2945-428d-bb76-bed09448eff5-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:47:41.848893 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:41.848852 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" path="/var/lib/kubelet/pods/3ebf9f4a-2945-428d-bb76-bed09448eff5/volumes" Jul 15 20:47:46.447384 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:46.447345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:51.130075 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:51.130031 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:57.093353 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:57.093311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:47:57.159628 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:47:57.159591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:48:01.778794 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:01.778753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:48:01.779230 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:01.779087 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" containerID="cri-o://155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4" gracePeriod=30 Jul 15 20:48:01.803808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:01.803776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:48:02.025768 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:02.025732 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:48:02.290593 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:02.290552 2576 patch_prober.go:28] interesting pod/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 20:48:02.290773 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:02.290643 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 20:48:02.455768 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:02.455729 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8c84d6fd-vkc4x_2d5f6214-b679-4d39-8b6c-1c69c8e544cc/authorino/0.log" Jul 15 20:48:07.050588 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.050509 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:48:07.173746 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173710 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.173746 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173780 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173804 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173838 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wf5f\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174050 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.173999 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174082 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket\") pod \"3e167b89-26ad-4ab7-892a-85816f906eeb\" (UID: \"3e167b89-26ad-4ab7-892a-85816f906eeb\") " Jul 15 20:48:07.174329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:48:07.174329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:48:07.174329 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174315 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:48:07.174556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174474 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.174556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174492 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.174556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174507 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e167b89-26ad-4ab7-892a-85816f906eeb-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.174556 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174503 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:48:07.174920 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.174895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:48:07.176467 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.176438 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 20:48:07.176596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.176450 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:48:07.176596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.176549 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:48:07.176596 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.176561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f" (OuterVolumeSpecName: "kube-api-access-8wf5f") pod "3e167b89-26ad-4ab7-892a-85816f906eeb" (UID: "3e167b89-26ad-4ab7-892a-85816f906eeb"). InnerVolumeSpecName "kube-api-access-8wf5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:48:07.204658 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.204622 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerID="155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4" exitCode=0 Jul 15 20:48:07.204833 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.204668 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" Jul 15 20:48:07.204833 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.204690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerDied","Data":"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4"} Jul 15 20:48:07.204833 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.204742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx" event={"ID":"3e167b89-26ad-4ab7-892a-85816f906eeb","Type":"ContainerDied","Data":"ce20311ee59470f75784c2e76c10515a25b1c6a098538ee44e3d712f74f33b53"} Jul 15 20:48:07.204833 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.204768 2576 scope.go:117] "RemoveContainer" containerID="155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4" Jul 15 20:48:07.214404 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.214384 2576 scope.go:117] "RemoveContainer" containerID="8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195" Jul 15 20:48:07.222996 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.222972 2576 scope.go:117] "RemoveContainer" containerID="155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4" Jul 15 20:48:07.223357 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:48:07.223310 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4\": container with ID starting with 155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4 not found: ID does not exist" containerID="155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4" Jul 15 20:48:07.223357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.223341 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4"} err="failed to get container status \"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4\": rpc error: code = NotFound desc = could not find container \"155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4\": container with ID starting with 155f38fd8d70dc5d545f91d973742ab16f2b0a9ab4caf1784f9ca7f390005cc4 not found: ID does not exist" Jul 15 20:48:07.223357 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.223362 2576 scope.go:117] "RemoveContainer" containerID="8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195" Jul 15 20:48:07.223673 ip-10-0-141-87 kubenswrapper[2576]: E0715 20:48:07.223651 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195\": container with ID starting with 8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195 not found: ID does not exist" containerID="8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195" Jul 15 20:48:07.223729 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.223680 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195"} err="failed to get container status \"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195\": rpc error: code = NotFound desc = could not find container \"8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195\": container with ID starting with 8946fde3a843d6b1f6af56cc7c402044f100c2ba64ce2c336db8495384237195 not found: ID does not exist" Jul 15 20:48:07.229184 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.229146 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:48:07.235105 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.235076 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5292f5-openshift-default-585b6c69c5-h7fdx"] Jul 15 20:48:07.275808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275774 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.275808 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275806 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.275959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275816 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wf5f\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-kube-api-access-8wf5f\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.275959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275826 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.275959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275836 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.275959 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.275845 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e167b89-26ad-4ab7-892a-85816f906eeb-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\"" Jul 15 20:48:07.720795 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.720763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxm8n_66131016-f44c-4f4b-bc91-bdc733e13185/manager/0.log" Jul 15 20:48:07.845957 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.845923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8m5rm_2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2/manager/0.log" Jul 15 20:48:07.849031 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:07.848991 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" path="/var/lib/kubelet/pods/3e167b89-26ad-4ab7-892a-85816f906eeb/volumes" Jul 15 20:48:08.090062 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:08.090000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ht6rv_e7a239d1-3a23-44fc-95c5-384d5d7490a1/server/0.log" Jul 15 20:48:08.214897 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:08.214866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2xmqq_aa76a2a8-8617-4d30-930d-44b2d8f4b9a8/manager/0.log" Jul 15 20:48:08.469560 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:08.469476 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f6d85d69-9qhw8_1c8ba5a7-509f-445f-a8d7-0d46952bc305/manager/0.log" Jul 15 20:48:08.476633 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:08.476565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f6d85d69-9qhw8_1c8ba5a7-509f-445f-a8d7-0d46952bc305/copy-manifests/0.log" Jul 15 20:48:10.690887 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:10.690855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tbwqw_4cfed204-d523-4d95-95da-0762ef9dbb27/postgres/0.log" Jul 15 20:48:11.948494 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:11.948464 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8c84d6fd-vkc4x_2d5f6214-b679-4d39-8b6c-1c69c8e544cc/authorino/0.log" Jul 15 20:48:12.277803 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:12.277725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z4pc8_e7b4579e-b81c-41b9-9b3c-4f5793da39f1/kuadrant-console-plugin/0.log" Jul 15 20:48:12.401607 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:12.401573 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cslc6_afb2e1b7-f79f-4d44-879f-fc3d4a563b75/registry-server/0.log" Jul 15 20:48:12.524425 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:12.524387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dbjcd_e1856216-e281-406e-837f-1be7b670af2a/manager/0.log" Jul 15 20:48:12.758802 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:12.758768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6stj9_93af46a2-d6f3-444b-8844-2dcd6f82d622/manager/0.log" Jul 15 20:48:13.107826 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:13.107793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4fpxhq_d9eca3c8-6b0c-4314-b73e-6cd47143443f/istio-proxy/0.log" Jul 15 20:48:13.458491 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:13.458402 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dcf7fc97d-r4hd9_86dddc3b-e6d9-45ba-9014-3233d427e6c3/kube-auth-proxy/0.log" Jul 15 20:48:13.582452 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:13.582423 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/1.log" Jul 15 20:48:13.827354 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:13.827260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-77fnq_b5b47d8d-f9ff-4431-8210-a1663fc15d26/payload-processing/0.log" Jul 15 20:48:14.276673 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:14.276642 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x42kq_8427d260-d1c2-4e8f-a3be-873f5efd57cc/main/0.log" Jul 15 20:48:14.841968 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:14.841928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-ttjks_4d4459b6-3662-4341-875b-ed572200e9a2/main/0.log" Jul 15 20:48:20.395542 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:20.395497 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:48:20.496030 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:20.495971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" limit=1024 Jul 15 20:48:23.336223 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:23.336189 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dfgv2_5ccd8254-4a7f-43c7-aee8-bd36cd411958/global-pull-secret-syncer/0.log" Jul 15 20:48:23.477254 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:23.477224 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7h9pp_78e336f2-a47f-4429-abab-4d52971db61b/konnectivity-agent/0.log" Jul 15 20:48:23.607976 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:23.607851 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-87.ec2.internal_152de02e452d1b71aa2bf2bbb50373e3/haproxy/0.log" Jul 15 20:48:29.123619 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:29.123584 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8c84d6fd-vkc4x_2d5f6214-b679-4d39-8b6c-1c69c8e544cc/authorino/0.log" Jul 15 20:48:29.202503 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:29.202475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z4pc8_e7b4579e-b81c-41b9-9b3c-4f5793da39f1/kuadrant-console-plugin/0.log" Jul 15 20:48:29.258239 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:29.258201 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cslc6_afb2e1b7-f79f-4d44-879f-fc3d4a563b75/registry-server/0.log" Jul 15 20:48:29.442652 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:29.442563 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dbjcd_e1856216-e281-406e-837f-1be7b670af2a/manager/0.log" Jul 15 20:48:29.567526 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:29.567496 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6stj9_93af46a2-d6f3-444b-8844-2dcd6f82d622/manager/0.log" Jul 15 20:48:31.661846 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:31.661813 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qwnw_07c11d81-3bf5-4022-9a88-e4a2a3de4a7f/node-exporter/0.log" Jul 15 20:48:31.682521 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:31.682498 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qwnw_07c11d81-3bf5-4022-9a88-e4a2a3de4a7f/kube-rbac-proxy/0.log" Jul 15 20:48:31.703269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:31.703232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qwnw_07c11d81-3bf5-4022-9a88-e4a2a3de4a7f/init-textfile/0.log" Jul 15 20:48:33.509070 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509031 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn"] Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509380 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509412 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509419 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509429 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509434 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55dbdc17-9447-4717-832a-8c3a0afe0504" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509442 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509449 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.509455 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509458 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509463 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509513 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509520 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e167b89-26ad-4ab7-892a-85816f906eeb" containerName="istio-proxy" Jul 15 20:48:33.509743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509529 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.509743 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.509535 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebf9f4a-2945-428d-bb76-bed09448eff5" containerName="istio-proxy" Jul 15 20:48:33.512533 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.512511 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.514950 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.514931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhp28\"/\"openshift-service-ca.crt\"" Jul 15 20:48:33.515092 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.514988 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhp28\"/\"kube-root-ca.crt\"" Jul 15 20:48:33.516067 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.516048 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhp28\"/\"default-dockercfg-kjwc9\"" Jul 15 20:48:33.524224 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.524201 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn"] Jul 15 20:48:33.611769 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.611730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskx6\" (UniqueName: \"kubernetes.io/projected/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-kube-api-access-hskx6\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.611769 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.611770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-proc\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.612042 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.611798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-lib-modules\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.612042 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.611888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-sys\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.612042 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.611929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-podres\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713257 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-proc\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-lib-modules\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-sys\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-podres\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-proc\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskx6\" (UniqueName: \"kubernetes.io/projected/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-kube-api-access-hskx6\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713437 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-sys\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713642 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-lib-modules\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.713642 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.713470 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-podres\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.722225 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.722200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskx6\" (UniqueName: \"kubernetes.io/projected/c4c90ef3-52bf-4eaa-8d76-d92a566df51f-kube-api-access-hskx6\") pod \"perf-node-gather-daemonset-vmbnn\" (UID: \"c4c90ef3-52bf-4eaa-8d76-d92a566df51f\") " pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.823042 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.822934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:33.956297 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:33.956230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn"] Jul 15 20:48:33.958688 ip-10-0-141-87 kubenswrapper[2576]: W0715 20:48:33.958662 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4c90ef3_52bf_4eaa_8d76_d92a566df51f.slice/crio-09e61a6dcab446322658a6f1a060a3e408556d0d7a20b6f239c49b58b00001e0 WatchSource:0}: Error finding container 09e61a6dcab446322658a6f1a060a3e408556d0d7a20b6f239c49b58b00001e0: Status 404 returned error can't find the container with id 09e61a6dcab446322658a6f1a060a3e408556d0d7a20b6f239c49b58b00001e0 Jul 15 20:48:34.335937 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:34.335897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" event={"ID":"c4c90ef3-52bf-4eaa-8d76-d92a566df51f","Type":"ContainerStarted","Data":"b1ca783fac37fe68b4001ab3ee247653952dba774af2d7bc4520c0985158740b"} Jul 15 20:48:34.336133 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:34.335987 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:34.336133 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:34.336004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" event={"ID":"c4c90ef3-52bf-4eaa-8d76-d92a566df51f","Type":"ContainerStarted","Data":"09e61a6dcab446322658a6f1a060a3e408556d0d7a20b6f239c49b58b00001e0"} Jul 15 20:48:34.355362 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:34.355268 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" podStartSLOduration=1.355250907 podStartE2EDuration="1.355250907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:48:34.353914309 +0000 UTC m=+2991.063480560" watchObservedRunningTime="2026-07-15 20:48:34.355250907 +0000 UTC m=+2991.064817158" Jul 15 20:48:36.289937 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:36.289899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vt4vk_d1c8a497-4d8b-49a3-843f-16d6445ded03/dns/0.log" Jul 15 20:48:36.319164 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:36.319115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vt4vk_d1c8a497-4d8b-49a3-843f-16d6445ded03/kube-rbac-proxy/0.log" Jul 15 20:48:36.444551 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:36.444521 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tx46d_8c00e0e0-0d9e-4d41-a987-b0d7a8dc51cd/dns-node-resolver/0.log" Jul 15 20:48:36.982777 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:36.982745 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6clq5_bf7ab018-02e3-4dd9-bd83-26c6719e02c7/node-ca/0.log" Jul 15 20:48:37.983462 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:37.983429 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4fpxhq_d9eca3c8-6b0c-4314-b73e-6cd47143443f/istio-proxy/0.log" Jul 15 20:48:38.192371 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:38.192343 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dcf7fc97d-r4hd9_86dddc3b-e6d9-45ba-9014-3233d427e6c3/kube-auth-proxy/0.log" Jul 15 20:48:38.228269 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:38.228235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/0.log" Jul 15 20:48:38.282645 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:38.282546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/1.log" Jul 15 20:48:38.365564 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:38.365522 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-77fnq_b5b47d8d-f9ff-4431-8210-a1663fc15d26/payload-processing/0.log" Jul 15 20:48:39.103751 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:39.103718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8b6nk_0192199e-9b55-451a-8d41-bb9d2a5549bb/serve-healthcheck-canary/0.log" Jul 15 20:48:39.720273 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:39.720241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fbk4_fcf59bdb-94a6-4b31-a825-9c7c1bdb163f/kube-rbac-proxy/0.log" Jul 15 20:48:39.741979 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:39.741957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fbk4_fcf59bdb-94a6-4b31-a825-9c7c1bdb163f/exporter/0.log" Jul 15 20:48:39.764791 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:39.764772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6fbk4_fcf59bdb-94a6-4b31-a825-9c7c1bdb163f/extractor/0.log" Jul 15 20:48:40.352589 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:40.352555 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhp28/perf-node-gather-daemonset-vmbnn" Jul 15 20:48:42.199470 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.199431 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zxm8n_66131016-f44c-4f4b-bc91-bdc733e13185/manager/0.log" Jul 15 20:48:42.446609 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.446505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8m5rm_2af7ed50-bdbe-4fef-bbcc-1df601c9c9c2/manager/0.log" Jul 15 20:48:42.579823 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.579796 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ht6rv_e7a239d1-3a23-44fc-95c5-384d5d7490a1/server/0.log" Jul 15 20:48:42.692407 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.692376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2xmqq_aa76a2a8-8617-4d30-930d-44b2d8f4b9a8/manager/0.log" Jul 15 20:48:42.812281 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.812176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f6d85d69-9qhw8_1c8ba5a7-509f-445f-a8d7-0d46952bc305/manager/0.log" Jul 15 20:48:42.833322 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:42.833278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56f6d85d69-9qhw8_1c8ba5a7-509f-445f-a8d7-0d46952bc305/copy-manifests/0.log" Jul 15 20:48:43.921597 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:43.921569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:48:43.929783 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:43.929759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dkmdg_069a64e5-5074-4274-8b9e-020eeb44415a/ovn-acl-logging/0.log" Jul 15 20:48:44.068448 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.068348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4fpxhq_d9eca3c8-6b0c-4314-b73e-6cd47143443f/istio-proxy/0.log" Jul 15 20:48:44.228434 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.228407 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dcf7fc97d-r4hd9_86dddc3b-e6d9-45ba-9014-3233d427e6c3/kube-auth-proxy/0.log" Jul 15 20:48:44.254537 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.254494 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/0.log" Jul 15 20:48:44.311211 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.311178 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/1.log" Jul 15 20:48:44.313418 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.313393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s6n9s_6b67276a-a46a-470e-b64c-1d262b58dd62/openshift-lws-operator/0.log" Jul 15 20:48:44.359429 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:44.359397 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-77fnq_b5b47d8d-f9ff-4431-8210-a1663fc15d26/payload-processing/0.log" Jul 15 20:48:45.636003 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:45.635944 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4fpxhq_d9eca3c8-6b0c-4314-b73e-6cd47143443f/istio-proxy/0.log" Jul 15 20:48:45.817435 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:45.817403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dcf7fc97d-r4hd9_86dddc3b-e6d9-45ba-9014-3233d427e6c3/kube-auth-proxy/0.log" Jul 15 20:48:45.843439 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:45.843408 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/0.log" Jul 15 20:48:45.902142 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:45.902051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-8xbb4_e0f6fb60-49fc-42c0-8328-bac5525298d1/istio-proxy/1.log" Jul 15 20:48:45.955300 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:45.955264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-77fnq_b5b47d8d-f9ff-4431-8210-a1663fc15d26/payload-processing/0.log" Jul 15 20:48:49.511631 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:49.511599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-pbvvk_458b315f-8825-4a60-894f-225074e42b5d/migrator/0.log" Jul 15 20:48:49.535789 ip-10-0-141-87 kubenswrapper[2576]: I0715 20:48:49.535758 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-pbvvk_458b315f-8825-4a60-894f-225074e42b5d/graceful-termination/0.log"